According to this web page on the US Customs and Border Patrol web site (last updated March 2014):
Can I cross the border via foot, vehicle, or air without visiting an official port of entry?
If you are entering the U.S. through the Northern or Southern border, you must enter through an open Port of Entry staffed with CBP Officers. The only exceptions to this would be if you are entering with automatic I-68 program status, in which case your yearly registration and telephonic notification of entry is sufficient. There are currently no programs in place for individuals entering the U.S. by foot (i.e. backpackers, hikers, jogger's etc.) or air that exempts them from this entry requirement.
Failure to enter through an open staffed port is considered an illegal entry and can result to you being fined and removed from the United States.
There is no regulation that requires a traveler departing the U.S. to exit through an open port of entry staff with CBP Officers. However, if you are exporting goods from the U.S. (i.e. a car or vehicle) you are required to depart through a port of entry, so a CBP Officer can verify you are the owner of the vehicle. Also, if leaving the U.S. by land, both Mexico and Canada require you to enter their countries through official ports of entry, so although you will not necessarily be breaking U.S. law to exit the U.S. anywhere you wish, you will immediately run afoul of Mexican or Canadian law if you do not enter through a legal port of entry.
So this would be illegal. (The automatic I-68 program deals only with entry by boat.)
If you discover you have crossed it by mistake, then presumably you should immediately get back into the country where you started, as quickly as possible. It is possible (perhaps even likely) that you would be stopped by border guards first, in which case you would have to hope you could convince them not to arrest you.
If you actually want to cross the border, then once back on your original side of the border, you should proceed to an official port of entry.