This answer assumes that you are holding either Article 10 or 20 residence card.
As a holder of residence card either under Article 10 or 20 of directive 2004/38 known as "Free Movement Directive" you are indeed free to travel to any EEA member country as long as your EEA family member, in this case your wife, accompanies you and you have appropriate documents with you. What you need to bring is evidence that you have the right of admission under EU law, which means that you must bring with you:
- your residence card
- your passport
- evidence of your ties to an EEA national you are travelling with, in your case that will be a marriage certificate
It is also worth adding that you can travel without being accompanied by a family member from EEA as long as you can bring supporting evidence that this person is lawfully in the UK already.
To also clarify some of your worries, the police is usually not involved in immigration matters - the border security is. And if they think that there is something not right they may indeed detain and question you to clarify anything they may find suspicious. That is also why it's best to apply for the family permit well before travelling, as application costs nothing and it reduces the possibility of that happening.
With a bit of luck your family permit will arrive before you depart, but if not then as long as your situation is genuine and you carry the required documents you shouldn't have any problems at the border.