Of course, why not? Note that a somewhat related, and much more common, situation does happen regularly as people move around with private crafts. In that case, it's up to you to follow all relevant rules and to report to the authorities within a certain time (usually it's understood you can at least dock to the harbor and walk up to the relevant office so you have already set foot in the country before getting clearance but you shouldn't wander around).
In a way, you could consider border checkpoints and all other enforcement measures as a convenience that helps travelers respect the rules but the mere fact that a border is not guarded does not exempt you from anything.
Where you come from during a particular trip also should not generally make any difference, countries typically define rules about entry, import duties, etc. independently of each other. While both typically happen in quick succession, the relevant event is entering the country, not leaving another one. Therefore, I see no reasons why you would be exempted by virtue of having transited through international waters.
Also, in many places where you would consider long-distance open-sea swimming (e.g. the English Channel), there are rules about safety and the like that would seem to be a more pressing concern (e.g. I believe than the French authorities do not allow swimming from their side so all recent crossings start from England). Realistically, you would secure authorization in advance and travel with an escort (a kayak to keep you company and a larger boat to make you visible, provide food, etc.) At the destination, authorities will be warned and might wait for you on the beach or wherever you plan to arrive.