It's hard to understand why this can be the case without some background in how airline fares work, so lets start with a bit of background...
For each route, airlines will publish a number of different fares, each with a distinct "fare code" - normally with weird names like "TANRKL", "YSJWNN" or "X34Q". Corresponding to these fare codes are "fare rules" which dictate things like whether changes/refunds are allowed on that fare class (and what fees are payable for them), whether stopovers are allowed, etc.
In addition to the customer-facing conditions the fare rules include controls over what type of trips that fare can be used for, with terms such as "One-way", "Return", "Open-jaw", "Circle", and "end-on-end" being common.
eg, looking at one specific "cheap" AA fare ("ONE0F0Q1") I can see the conditions "FARES GOVERNED BY THIS RULE CAN BE USED TO CREATE ROUND-TRIP/CIRCLE-TRIP/SINGLE OPEN-JAW JOURNEYS". A more expensive fare on the same route (fare code "H2") has the conditions "FARES GOVERNED BY THIS RULE CAN BE USED TO CREATE ONE-WAY JOURNEYS."
If you're booking a return fare then you will be able to use the cheaper fare, whereas if you book a one-way it'll automatically bump you up to the more expensive fare as the cheaper fare doesn't allow one-way. In each case the booking engine will automatically pick the cheapest fare for what you're doing - it's actually very possible (especially for domestic flights) that you'll end up with 2 "one-way-only" fares instead of a "return" fare if that's the cheapest way to do it.
Frequently, as you've said, the one-way ticket will actually cost more than it would to buy a return. Some people will tell you in these situations that you can just buy the return and not fly the 2nd leg - but it's worth keeping in mind that the one-way ticket, by being a higher fare class, probably includes better ticket conditions than the cheaper return. eg, it's likely that the cancellation fee on the more expensive ticket will be reduced (or even eliminated), and fees for making changes to the ticket will often be lower. ie, you are normally getting a "better" ticket for your money - even if it means that you end up sitting in exactly the same seat!
In addition to one-way and return there's a third class of travel, called "circle". A circle fare is when you fly AAA-BBB-CCC-AAA, without returning to BBB on the way back. As with one-way, some fares allow circle routes, and others do not - with the likelihood of them being allowed increasing with the price of the fare.
Next comes stopovers. Some tickets will allow you to have a "stopover" with wording something like this :
ORIGINATING UNITED KINGDOM -
1 FREE STOPOVER PERMITTED IN CONTIGUOUS U.S.A.
That means that on the way from AAA-CCC you can stop somewhere in the middle. The problem is that it needs to be somewhere "on the way". eg, if you're flying AAA-BBB-CCC, then you can have a stopover at BBB, but not anywhere else.
The other term you'll often see mentioned in the fare conditions is "end-on-end", with text something like this :
END-ON-END
END-ON-END COMBINATIONS PERMITTED. VALIDATE ADJACENT
LINE OF FLIGHT FARE COMPONENTS ONLY. TRAVEL MUST BE VIA
THE POINT OF COMBINATION.
"End-on-end" refers to booking 2 distinct fares, but having them combined on a single ticket. This can be used to give fares for routes that an airline has not published a fare for - but it can also be used to get cheaper fares if the combination of the two fares is cheaper. Some fares will also include text like "SIDE TRIPS PERMITTED." in the end-on-end rules, which means that not only can you add in another trip at your destination, but you can also add in a separate "side trip" at a stopover point.
As an example, I recently flew from San Francisco (SFO) to Newark (EWR). When I looked up the prices, it was around $800 to fly direct, or around $400 to fly via Denver (DEN). Digging into this, the reason was because the only fare available on SFO-EWR was a high-priced fare (United ticket class "H"). But on SFO-DEN and DEN-EWR there were cheaper fares available - around $200 for each leg, and both fares allowed "end-on-end" combinations. Thus I bought a single ticket SFO-DEN-EWR, but behind the scenes I was actually paying for 2 distinct legs, each with their own fare codes.
Now, back to your specific question. You're after LON-AUS-GIG-LON. There's a number of different ways that a ticket like this could be built.
Firstly, you could do a number of "one-way" fares - LON-AUS, AUS-GIG, GIG-LON. This will always work (as airlines always allow one-way fares), but as mentioned above it'll likely be expensive.
Next, you could do it as a "circle" fare. To do that, you'd need to find one airline that had published fares for each of LON-AUS, AUS-GIG, GIG-LON, AND those fares allowed "circle" trips.
Alternatively you could do it as 2 end-on-end return fares - LON-AUS-LON and AUS-GIG-AUS. The requirement for circle trips is now gone, as it's just 2 simple return flights, but you're having to backtrack to AUS on the return as end-on-end fares normally include the condition that "TRAVEL MUST BE VIA THE POINT OF COMBINATION", which in this case means you need to pass via AUS in both directions.
The next option is to use a stopover. eg, you could potentially fly a single fare of LON-GIG-LON, with a stopover of AUS on the way over. The problem is that there's no airlines that have a flight from LON-GIG that stops in AUS so in this case that's not going to work.
The final option - and the reason that it's best to leave finding fares to computers rather than humans - is to combine one or more of the options above to find the best route/price.
Plugging your requirements into ITA Software's search engine for some random dates in February it came up with a best price of £691.89 on American Airlines. Looking at the details of how it came up with that price, we see that it actually used 4 different fare code to build the entire fare :
Fare 1: Carrier AA ONE0F0Q1 LON to RIO (rules)
Passenger type ADT, round trip fare, booking code O
Covers LHR-JFK (Economy), JFK-GIG (Economy)
Fare 2: Carrier AA QA21ERP1 NYC to AUS (rules)
Passenger type ADT, one-way fare, booking code Q
Covers JFK-AUS (Economy)
Fare 3: Carrier AA QA21ERP1 AUS to NYC (rules)
Passenger type ADT, one-way fare, booking code Q
Covers AUS-JFK (Economy)
Fare 4: Carrier AA ONE0F0Q1 RIO to LON (rules)
Covers GIG-JFK (Economy), JFK-LHR (Economy)
Passenger type ADT, round trip fare, booking code O
So it's actually booked one round-trip flight LON-GIG-LON (fare 1 and 4), with a stopover in JFK on the way over. If we were to check the fare rules we'd find that fare allowed "return" trips, but almost certainly doesn't allow "one-way" trips. We also know that it must allow stopovers (as we've got one in JFK).
Next we've got 2 one-way fares from NYC-AUS (fare 2) and AUS-NYC (fare 3). These are domestic fares in the US, and in most cases airlines don't bother with the whole one-way/return distinctions for domestic fares, so it's not more expensive to book them as one-way rather than return.
The trip to AUS is occurring from your stopover point (JFK) of the first trip rather than your destination (GIG), which means that the fare for LON-GIG-LON allowed a "side trip" during the stopover in JFK.
Putting this all together, you get your full trip - LHR-JFK-AUS, AUS-JFK-GIG, GIG-JFK-LHR for under 700 quid (around US$1100).