We want to travel from London to Peppa Pig World in Paultons Park. From enquiring, the fastest and most cost-effective way to reach it is by taking a train from London to Southhampton and then taking a taxi to the park. I checked about buying a ticket in advance on TrainLine.com and I see that I need to book a specific train (meaning a specific time). On the way there it's not really a problem, but the way is problematic, it would be very hard to set a schedule and stick to it when travelling to an amusement park with small children, they might not want to leave at the planned time, or be tired and want to leave way earlier. Ordering an open return ticket costs more that 100 GBP. Do I have to book the return (and even the forth ticket) in advance? How much will a train ticket cost if I buy it in the station (for adults and children)? How far ahead do I need to buy the ticket? Can I book it when I see that we are finished and are on our way back?
Do I have to book the return (and even the forth ticket) in advance?
No. In Great Britain, there are two main types of ticket — Advance tickets, which must be booked in advance, are always single tickets, and tie you to one specific train; and "walk-up" tickets (mostly (Super) Off-Peak and Anytime Singles and Returns), which are flexible and allow you to travel on any train (possibly subject to restrictions on travelling in the rush hour), and can be bought at a station right before travel. If you can fix both your outward and return trips you might find it cheaper to buy two Advance tickets, which must be booked in advance, but if not, a walk-up single is almost the same price as a walk-up Return (out and back) ticket, so it makes sense just to buy the flexible walk-up Return (out and back) tickets. In that case, you can just buy them at Waterloo station (or any other National Rail station) on the day of travel; there's no cost saving to buying walk-up tickets in advance.
Note that South Western Railway don't do seat reservations at all, so there's no advantage whatsoever to buying a walk-up ticket ahead of time.
How much will a train ticket cost if I buy it in the station (for adults and children)?
Without a railcard
Anytime Day Return (no time restrictions): £47.20 per adult or £23.60 per child
Off-Peak Day Return (valid at any time at weekends, or any time after 09:00 leaving Waterloo on a weekday with no relevant restriction coming back): £44.10 per adult or £22.05 per child
Weekend Super Off-Peak Day Return (valid on weekends only, and only leaving Waterloo 11:00 or later, with no relevant restrictions coming back): £38.40 per adult or £19.20 per child
With a railcard
You can buy a Family & Friends Railcard, which is only valid when travelling with children, for £30. This gives you 1/3 off the adult fares listed above for up to four adults, and 60% off the children's fares for up to four children. I would be surprised if this doesn't work out as worth the cost of the one-year railcard just for this one trip. You can buy a railcard at a station with a staffed ticket office (for instance, London Waterloo), or online through the Family & Friends Railcard website. Once you've got a railcard you can buy railcard discounted tickets from ticket offices and ticket vending machines.
Note that this railcard is not valid for travel in the morning peak on a weekday (specifically it's not valid in the morning on journeys within the Network Area until the Off-peak Day Return is valid), so if you're travelling on a weekday you would need to leave after 09:00 if you're taking this option.
This answer so far has spoken about the fast route to Southampton from London Waterloo on South Western Railway trains. One more thing worth mentioning is that there's another route to Southampton Central, on a Southern train out of London Victoria, which is much cheaper — but much, much slower (two and a half hours from Victoria vs one and a quarter from Waterloo). You probably don't want to go this way (if you cared that much about saving money I'd imagine you'd be getting a coach anyway), but I thought I'd mention it anyway. The tickets valid on both faster and slower routes are routed "ANY PERMITTED" (this is what you'll see when buying them from a ticket machine); the ones valid on the slower route only are routed "THREE BRIDGES".
How far ahead do I need to buy the ticket?
This is answered above — basically, you don't.
Can I book it when I see that we are finished and are on our way back?
If you're talking about Advance fares — no. It varies between train operators, but with South Western Railway who operate the faster trains between London and Southampton, you can only buy Advances up to 23:59 on the day before travel (and yes, apparently this applies to mobile eTickets too — see the terms and conditions on that page).
Assuming you are returning the same day, an Anytime Day Return ticket from London to Southampton costs £47.20 for an adult and £23.60 for a child. This ticket is fully flexible and can be used for one journey from London to Southampton and back on the specified day. This ticket can be purchased in advance or at the station on the day.
It would probably not make sense for you to pre-purchase a restricted Advance ticket on your outbound journey to Southampton, as a flexible one-way ticket from Southampton to London costs nearly as much as the return ticket above (£44.80 for an adult).
You've explained that you're travelling as a party of 2 adults and 2 children, and that the >£100 cost is for the four of you together. Unfortunately, this is likely to be the cheapest flexible fare available to you (assuming that this is for a return journey and not a one-way journey).
Advance fares (i.e. tickets marked "Advance", not all tickets purchased in advance of your journey) can be substantially cheaper than flexible fares as you've discovered, but they are extremely restrictive. In addition, on the London to Southampton route, they must be purchased at least one day before you travel, so you would not be able to take advantage of these savings were you to purchase on the day of your trip. The only tickets available to purchase for you on the day of your trip would be the flexible fares I've described above.
Another poster has mentioned the Family and Friends railcard. Please note that the Family and Friends railcard is not valid on journeys departing London to Southampton before 9:30 AM on weekdays, so do check that a discounted fare is available if you intend on purchasing a railcard.
That is a typical saving for booking on specific trains, well in advance (advance tickets are sold in limited numbers and do sell out).
You can save a fair bit with a family and friends railcard, but that costs money itself. If you end up buying the anytime day return, the railcard should pay for itself on this one trip. Mine did the first long distance journey I did with my daughter.
Also, under fives travel free, and if you're going to Peppa world there's a fair chance at least one of yours is under five.
Given that Advance tickets are sometimes a lot cheaper than an off-peak day return, you could consider:
Advance tickets for one particular train to Southampton; and then
Advance tickets for two or three different options on the way back.
Then, if you want to go home earlier, you use an earlier booked option (and not use the tickets for the later option(s) you have booked). If you want to go later, you use a later booked option (and not use the tickets for the earlier option(s)).