Are Allergies Rude?
Imagine you were allergic to, say, garlic. Would you feel rude to mention this to your host? In my opinion you should not. Similarly, should not feel rude when mentioning any other dietary restriction or preference, regardless of the reason behind it. It makes no sense to compare the importance of one dietary restriction over the other, nor does it make any sense to grade personal reasons on someone else's scale: my coriander allergy is not more important than your no-vegemite religious reason.
Plan and Inform Ahead
Personally, whenever I have dietary restrictions I mention them and people are usually most accommodating. In fact, what usually happens is that the host asks for dietary restrictions before I even mention them. In any case, be sure to mention your needs early enough to allow your host enough time to accommodate them by planning alternatives. Showing up to a dinner party, openly stating that you don't eat kiwis on Thursdays and spending the whole night fasting will definitely be perceived as rude.
If All Else Fails
If, upon stating your dietary preferences, you perceive unhappiness in your host then by all means change host. Why would you want to have dinner with someone with whom you are incompatible?