If I am on a tourist visa, and refused entry into the intended touring country, then who pays for the return ticket?
I would assume I will have to. But if I have no money of any kind, then what happens?
It depends both on local legislation (in the country you are denied entry) and the terms and conditions of the carrier bringing you there.
If you are travelling by air, the air line will of course check that you have all necessary travel documents before they let you board the flight at all, but if I understand your question correctly, you are asking what will happen if you are denied entry at the border even if your travel documents and visas are ok? Even if the air line has done everything in their power to check your eligibility to enter the destination country, they are in most cases still required by national law (in the destination country) to either bring you back to the origin country or if you are not eligible to reenter the origin country, to bring you anywhere else.
If you are travelling on a return ticket, most airlines are fair enough to let you use your return ticket for the unexpected return flight. Aside from that, most air lines regulate this in their terms of carriage and hold the passenger liable for any further costs.
Just as an example, here are Lufthansa's terms regarding denied entry. But as I said, you will find similar regulations in other air lines' terms of carriage as well:
Refusal of Entry 13.3. If you are denied entry into any country, you will be responsible to pay any fine or charge assessed against us by the Government concerned and for the cost of transporting you from that country. We may apply to the payment of such fare any funds paid to us for unused carriage, or any funds of the passenger in the possession of us. The fare collected for carriage to the point of refusal of entry or deportation will not be refunded by us.
If you don't have any money to pay these charges up front, the air line is still liable to transport you, but you must expect the air line to use any possible legal means to get the money back from you later.
It is sometimes possible to stay in the airport long enough to catch a flight to somewhere else, depending on the view of the immigration officer. Obviously, in such a case, you would pay for the ticket yourself, and I imagine your airline would support your petition. But if you mean a simple "No passport/visa? You're not getting past this desk.", then the conventions state that it is your carrier's responsibility to return you to whence you came.
(I have always understood that this is the reason why you have to present your passport at the check-in desk. No country requires a passport to leave the country, but the airline wants to know that you will be admitted before they allow you on board.)
EDIT: The Warsaw Convention specified that in cases where a passenger is denied entry, it becomes the responsibility of the carrier to transport the passenger back to the starting pouint. It did not, so far as I know, specify whether a charge could be made for this. However, the point seems to have been overtaken by events; immigration law in most countries (the United States and United Kingdom, at least, have published theirs online) specifies that if a passenger (sea or air) is denied entry, the carrier is liable not only for the fare but for a large fine as well, unless the passenger has misled them. In practice this means you will be transported back, and the airline will try to prove everything was your fault. Now is the time you find out what sort of legal advice your travel insurance provides.
If you are refused entry and came by air, the airline that brought you there has to take you back. Depending on the local law, it might also be fined if you did not have a visa and it failed to check. The airline might try to recover the funds from you later but nobody is paying for a ticket.
If you are refused entry on a land border, nobody will pay for a ticket either, you are just unable to enter the country and left stranded wherever you are. In some cases, when the control does not happen at the border itself but further inland (either on board a train or at a station), I have seen people forced to take a train in the other direction but I don't know what the rules were.
If for some reason you can't return where you came from (say you don't have the right to reenter the country you just left), the only thing left is to deport you somewhere else. Reasonable countries will at least try to deport you to a country from which you are a citizen. In that case, the country that wants to deport you pays for your ticket (and if necessary those of your police escort). Again, it might also impose a fine and try to recover money from you later but I suspect that most countries don't bother as many people who are deported will have very little money to begin with or come from countries where effective means to recover a fine are non-existent.
So in a nutshell, the country that sent you away or the airline that transported you might try to recover some money from you later on but it really does not matter if you have any at the moment you are denied entry. Either the airline will take care of the transport directly or it will be paid by the state removing you.
ICAO Annex 9 chapter 5 covers removal. Entries 5.10 and 5.11:
5.10 When a person is found inadmissible and is returned to the aircraft operator for transport away from the territory of the State, the aircraft operator shall not be precluded from recovering from such person any transportation costs involved in his removal. 5.11 The aircraft operator shall remove the inadmissible person to: a) the point where he commenced his journey; or b) to any place where he is admissible.
then who pays for the return ticket?
The IATA Ticketing Handbook sets out the procedure for inadmissible passengers in Section 126.96.36.199.
The Handbook is in copyright so I shall paraphrase, however it is not too difficult to find online if you want to read the original wording.
The final inbound airline is responsible for ticketing the inadmissible passenger to whatever place advised by the authorities. It should obtain payment in the following order:
If the passenger already has an outbound ticket (on any carrier) to the place specified by the authorities, then that ticket may be used immediately. Any restrictions preventing immediate use of the ticket (such as minimum stay, change fees, restriction to a specific airline, etc) should be ignored. The ticket should be revalidated (changed) by the final inbound airline and imprinted with the words "RESTRICTIONS WAIVED DUE TO INAD" in the endorsements box (to explain the situation to other carriers).
If there are unflown flight coupons remaining on the inbound ticket, the final inbound airline may appropriate these flight coupons and use their value towards payment of a new outbound ticket. (Again, the ticketing airline may disregard any endorsements or restrictions on those coupons.) The word "INAD" must be appended to the passenger's name on the new ticket.
If the passenger does not have a return ticket or the value of the unflown coupons does not cover the transportation costs, then the final inbound carrier is responsible for collecting the fare due for the outbound journey from the passenger.
If the final inbound carrier is unable to collect the fare due for the outbound journey from the passenger, that carrier is still responsible for issuing the outbound ticket. The cost of the outbound ticket is shared between all the carriers who offered carriage on the continuous journey from the point of origin on the ticket (or the last stopover, if there was one), to the place where entry was refused. The sharing is pro-rated by mileage, so the final carrier won't pay much if it just operated a short leg on a much longer itinerary.
However, if the final inbound carrier was technically incapable of issuing the onward ticket, then another inbound carrier may issue it instead; but the final inbound carrier is then responsible to the ticketing carrier for the full transportation cost (it isn't shared).