It is difficult and entirely depends where you want to go with your collection, I think the main reason the powerhouses get to the stage of micro varieties is because the initial list was filled, then the subsequent additions get filled, then when running out of things to collect, start to really scrutinize every coin, looking for differences.
At the moment I collect (pennies) all the years, mint marks, the main recognised varieties like 1903 open 3, 1911 Gouby X, recessed ear varieties etc, and then have a side collection for anything of interest. There are some I have like the missing wave varieties that I don't want to include in my collection because they're caused by the natural process of minting coins and not by purposeful design changes.
If you haven't already, I'd definitely recommend getting literature on your area of collecting because there are some excellent books out there to help you along the way.