I hold a US passport. If I enter and stay in a Schengen country for more than 3 months, i.e. if I overstay my Schengen visa by 6 months or more, and then wish to travel to and enter another Schengen country by rail or ship, will I be barred from entering, or made to pay a fine before entering? What possible penalties would I face?
As noted in the comments, the specifics depend on where you're found, because the penalties are defined in national law. That said, some general statements are possible that you may find useful.
if you are found in the Schengen area having been there for six or more months you will very likely be required to leave.
you may be allowed to leave on your own, being given a short time to arrange this, maybe a couple of days or if you're lucky a week or two. Otherwise you'd be detained and forcibly deported.
you would likely be fined. I have little idea how much, but I'd guess somewhere between several hundred euros and a few thousand euros.
you would likely be banned from returning to the Schengen area for some length of time. The ban should not be for longer than 5 years unless you are found to be a "serious threat."
you are unlikely to be discovered while crossing internal Schengen borders, but it's possible. It's also possible to be discovered by any police officer anywhere (if the conditions exist under national law for the the police officer to verify your immigration status).
if you are not discovered inside the Schengen area you will almost certainly be discovered as you leave. In this case you would probably be allowed to continue to leave, but you would still likely be fined and banned. This would take some time, so be sure to get to the airport a few hours early so you don't miss your flight.
Some Schengen countries readily grant national visas for extended stay as a tourist. Applying for such a visa is probably a much better way to go if you want to spend several months in the Schengen area and you can arrange your plans to be based in one of those countries. Otherwise, you may want to plan to spend two periods of just under 90 days separated by at least 90 days (a pattern that you can repeat indefinitely).
Another thing to consider is whether your presence in any Schengen country causes you to become a tax resident under that country's law. In many countries, this happens after 183 days regardless of your immigration status.