Is it possible if she is not Iranian to swim in her swimsuit or bikini in the sea?
She is not from Iran, she is European as me.
Is there some "police" that watches how people dress on the beach?
According to the article in the Global Post it is possible as long as she's in a female only area and hence you can't be with her. This information obviously is a little dated but the only area that is known for laxer regulations would be the island of Kish but that information is even more dated.
The most recent English news I was able to find is an increase in religious police so legally it is highly unlikely that regulations enforced for hijab would be laxer for bikinis.
The simple answer is no! If you are thinking of the Caspian sea, there may not be any "police" watching you per se, but the people of the North are quite religious, and they police themselves. Bikinis are out of question. If her scarf flies away, they will give you friendly warnings. Don't be scared, they won't harm you, they just will tell you to keep your Hijab on or something like that.
In female only swimming areas she can swim by herself. These areas will be covered (not at top, just at the sides).
In any case, you should not swim in the Caspian sea, it is too polluted. Someone mentioned the Island of Kish being more relaxed. Yes, it is, but not to the point that she can swim with a bikini. The things people get away with in Kish are dancing in public, for example, or singing music that is considered illegal in other parts of Iran, and less policing on Hijab, etc. But again, bikinis are out of question.
According to local rules, women cannot swim in bikini in presence of men around. They are limited places for women to swim in the sea which men are not allowed to enter there. However as a Iranian i recommend you to visit here since there are many other wonderful stuff which will inspire you :)
It is possible to the extent that she runs the risk of, at best, being deported from the country. However, though the piece below is four years old, it suggests it might be possible to find private beaches on the Caspian: