An Anytime Return London to Glasgow costs £365. The same route in Off-Peak costs £137.60.
An Anytime Rail&Sail Return London to Belfast via Cairnyran costs £108.
Rail travel from London to Cairnyran takes one through Glasgow, among other places.
Is the train ticket to and from Glasgow valid if I don't take the ferry?
There's not a lot they can do; they might figure out that I never appeared for the ferry and blacklist me, but that would be after the fact. The difference in price is very large and even the Anytime Rail&Sail is cheaper than the Off-Peak return, so for any train that can timetable-wise connect to or from a ferry, it should work. Similar tickets exist via Liverpool or Holyhead to Belfast or Dublin. The only additional restriction appears to be that the Rail&Sail ticket does not permit a break of journey, but that is a relatively minor restriction for most travellers.
(The advantage disappears for Advance travel.)