To accomodate the people living in/around Baarle-Nassau, there's bound to be some special rules between Belgium and the Netherlands, but I don't know them - but in general the rule is that a car should be registered in the country where it is used. The point is that taxes associated with the car are meant to contribute to the society that has expenses (e.g. maintainance of the roads) from it's use.
Of course there are rules for passing through and vacationing in other countries, but as soon your use doesn't match those, the rules get strict. An example:
I live and work in Copenhagen, but have colleagues who live in Malmø, Sweden. They are allowed to drive their Swedish registered cars to/from work, but (as far as I have understood) they are not allowed to use it for trips wholely within Denmark, i.e. they can go shopping in Danish stores on their way home from work.
When you have to stay in a different country for a limited amount of time there's a possibility to get permission for using your car while there. When I was 9 (so too young to care about the tax details), we lived in France for ~3 months, because my father had to attend some course together with some of his colleagues, some of which brought and used their Danish registered cars (my family didn't have a car) - they probably paid some tax to France (and possibly got some tax return in Denmark).
Returning to your cases.
The Dutch authorities were probably right when they issued that fine, and you should look into if your planned stay in the Netherlands is so long that you need a permission to bring and use your car while there.