Actually, the ones you've mentioned - they often have cheaper ways to get there, and in many cities the most common way is very cheap - it's not always expensive.
In your London example, for half the price of the Gatwick Express you can take Southern Rail to get to the airport. EasyJet has an EasyBus from Fulham Broadway which will take you there, and can cost you as little as two pounds!
To get to Heathrow, you can take the tube - a trip which takes an hour, for about 4 pounds! Or even less if you take two buses, and less if you have an Oyster card.
In Vancouver, it costs $3.75 to get to the airport on the Skytrain, from downtown. If you have a simple two-zone pass (like many locals do) it costs nothing extra.
Generally the high price comes down to a few key reasons:
- the cities you mention are expensive cities. The infrastructure, maintenance and so on costs a lot.
- 'tourist tax'. I say this tongue in cheek, but a lot of the costs can be mitigated with local know-how. The Sydney airport train costs so much because of the final charge for entering/exiting the airport - take a local bus to the second train station and you shave a chunk of the cost off.
- distance. Generally (with good city planning) the airports are a long distance from the city. As a result, the maintenance, fuel and time adds up. I refer you again to London - with a single Zone 1-2 tube ticket (a few quid) you can get to the London City Airport. Same city, less distance = less charge.
- profit. If an airport has a captive market, and they can, they could charge.
With regards to the weird taxi/shuttle thing - some people are worried about taking taxis in certain countries and may prefer being with others in a shuttle. Or they're paying for an 'executive' shuttle, for example. Shuttles could also be licensed, while you might get an unmarked taxi.
A good way to mitigate these in advance is to check out the Wikitravel page for the city in advance. Have a look at the 'Get in' section. It'll often describe several ways to get in from the airports, and the cost. For example, from EZE (international airport in Buenos Aires), you could take a taxi (costs!), a shuttle (medium), or the local bus (very little).