The cheapest trips involving multiple flight segments often have different airlines servicing the segments. Let's say a trip involved 3 flight segments, all using different airline companies. It would be tragic if someone was denied entry and all three airlines forwarded the fines to the passenger considering that the average fines per passenger are $3,500 and up to $10,000. Imagine suddenly owing $30,000 to 3 airlines!
When arriving at a country and you are refused entry at your final destination, immigration officials will ascertain what passport you used to travel, what airline you used and where else you would be admitted if they sent you there. Typically this is the departing airport from which you left at the start of your journey including all connecting flights.
For example, a flight leaving from UK to USA with a connection in Amsterdam will have the traveller returned to UK if they are refused entry at the US border. Of course, they may also send you directly to Amsterdam as they are certain you will be admitted here (since traveller in this hypothetical scenario holds a UK passport) and then how you get back home to the UK is entirely your problem. If you hold a round-trip ticket then the airline desk in Amsterdam are your best bet but this is entirely at the airline's discretion.
You will not pay any 'fines' as countries do not typically issue fines to those refused entry as this is completely unfair and does not truly reflect the situation. A traveller may present themselves at a border only to be told they are being refused entry because the immigration official does not believe their intentions. Yes, their intentions could be true but they won't be penalised for not properly presenting themselves at immigration nor being able to convince an immigration officer of their purpose of travel.
To answer your question directly - no, you will typically not have to pay any fines directly. Any return tickets you held will typically go towards the cost of sending you back home on an earlier flight. Obviously, being made to pay toward a cost of a flight and being refused solely for being refused entry are not the same thing.
Now, I cannot say for certain if airlines in the original departing country will face fines themselves for allowing a traveller who has a connecting flight without checking if they have the proper documents prior to departure, perhaps somebody who works in this particular field may wish to answer this part of the question. I have travelled from the UK to USA via Amsterdam and both at the UK departure airport they have checked my documents and again at Amsterdam to ensure I am carrying the correct documents. If they did not check my documents at the UK end then I am not sure if they would face a fine as they would not need to check my documents to travel to Amsterdam - and that is the location where I would be admitted.