That seems very true. There are some varieties - not the true rarities - that you should theoretically be able to, but in practice just cannot get in high grade. No matter how wealthy you may be, they just aren't coming up for sale. In many cases I would like to upgrade, but can't.
In the penny (currency) range, I'd cite:-
1843 (with or without colon)
1845 (only with great patience)
1849 (only with great patience, and at great cost)
1856 OT - never seen one for sale in high grade. Mine is fine only and no more than a gap filler.
1860 F 14, F16, F17 (got all three in high grade, from a number of years ago, but only one occasion for each in now 12 years, and opportunities taken, not seen any since).
1861 F18, F20, F25. (got all three, but the F20, fine only) - the F18 EF courtesy of a private sale by Ian Fall, and the F25 from the Hiram Brown sale Feb 2020 GEF with lustre (for £1200 hammer)
1864 plain 4 - got a VF specimen bought in 2012. Not seen a better one on offer since
1874 F68, F77, F78 (got all three, the F77 courtesy of a private sale by Ian Fall, the F78 GVF, and the F68 EF, but looks as though it's been scratched along the pavement under someone's foot. Would like to upgrade the F68 & F78, but again, they don't come up.
1879 narrow date - one only seen and bought from DNW September 2018.