I suspect it operates under the same system as for shipping to certain states in the US whereby sales tax is payable by the purchaser and is automatically added to their invoice, though as a seller, the money doesn't actually land in the vendor's account. As a seller, they won't charge the VAT personally, and let's face it, most sellers of coins are private individuals, many of whom don't believe VAT is due on any transaction, nor that HMRC should be in any way involved. Ebay are therefore acting as a tax collector at the point of import to get around this issue.
What's happened here is likely to be a half baked operating method, whereby ebay only apply 20% VAT and call it quits because the number of 5% items is minimal in the context of imports. I suspect the only way around it is to get payments out of ebay's control working again in whatever way is possible. You will lose buyer's protection (if it means anything), but given most people on both sides are honest, it should work for much of the time if the will is there. Making a separate Paypal payment outside eBay and ensuring the seller puts the right description on the customs label will work, but for low cost things, say up to about 75 quid, a 20% payment to ebay will be cheaper than the Royal Mail or courier admin fee plus the import vat which will become due if not already paid. Alternative is to veto it, because they are unlikely to listen to reasoned argument having already agreed terms with HM Gov. Ebay applying the right tax would be the cleanest and most equitable way forward, but don't hold your breath.
I'm also wondering how the Royal Mail will know that import tax has been paid? Their default position has to be charging tax unless there is proof of prior payment.
I don't know if the code that seems to be included in the address of every purchase is connected to this? Anybody?
Edit to add that if you are going to get 5% VAT applied on coins, then the vendor will somehow have to communicate the eligibility of the item for treatment to ebay. As we know, not all coins are equal, so differing rates apply and this will almost certainly be beyond the ken of a foreign vendor who is likely to be as au fait with tax codes and the finer points of the VAT regime as the man on the street is here. There is also another potential issue in that people may start describing items as those in lower tax bands and claim ignorance, which I'm sure HMRC have considered. Taxing at the top rate for imports and the buyer reclaiming excess tax is therefore likely to be the best option, as you will have both parties with the required knowledge and it protects all interests.