To be clear this is not a question on doing anything illegal. The question is about realms of control, and when you're "visiting" the territory of a country.
According to the UN's rules, territorial waters extend 12 nautical miles outside of the land. So to travel to country X, entering into those areas would put you under their territory.
What I'm wondering is whether a similar convention exists for altitude above ground/sea-level. I assume there's some limit, or the ISS would arguably be crossing countries' territories all the time...