As MJeffryes has noted in his answer, the prevailing interpretation of directive 2004/38/EC is that someone with a "Residence card of a family member of a Union citizen" may only use that card to benefit from the visa-free entry specified in Article 5(2) when they are traveling with or joining their EU family member. That condition is not explicit in the directive, and, as you note, it is an unreasonable or at least insurmountable barrier to the right of free movement for someone with a retained right of residence.
The directive, however, is not explicit on the question of whether a person enjoying a retained right of residence in one country continues to benefit from the other free movement provisions, such as a right of entry into another country. In other words, it provides that you retain a right of residence in the UK, but not necessarily that you retain a right of free movement throughout the EU.
To challenge the prevailing interpretation of the directive, you may have to go to court (I do not know whether the matter has already been decided by a court.) Before going to court, though, you might try lodging a complaint with SOLVIT, but they also link to the page that reflects the prevailing interpretation:
Your non-EU spouse, (grand)children or (grand)parents do not need to get a visa from the country they are travelling to if:
- They have an EU family member's residence card issued under EU rules by any EU country (except the country you are a national of), and they are travelling together with you or travelling to join you in another EU country. The residence card should clearly state that the holder is a family member of an EU national.