I am guessing that each airline has different luggage policy but I was wondering if there is a maximum luggage weight per passenger (assuming the passenger can pay the additional weight)?
If there is, can a very heavy (> 50kg) luggage be rejected?
Now you have clarified (in comments on the question) that you are interested in the mass range of baggage that is too heavy for the free allowance, but may still be carried without special arrangements, the policy may well vary from airline to airline - but probably not by much.
Charges are applied for bags over 23kg and up to 32kg. Bags over 32kg cannot be checked in and must be shipped separately as freight or cargo
which agrees with Relaxed's comment above.
There is no limit to the type of cargo an aircraft can carry. After all, the same 777 you use to fly a long haul flight is the one that is used to transport everything from aircraft engines to animals.
The restrictions that are in place are due to safety and prohibit carrying specific amounts or types of hazmat (hazardous material) like batteries which pose a safety risk.
The question comes in specifically with luggage; this is because unlike cargo, luggage is subject to some restrictions - mainly due to the existing infrastructure at airports that limits the type, size and weight of items that can be automatically processed.
In some areas, due to work safety issues - heavy luggage is not allowed as it may need to be manually loaded onto the aircraft.
I was wondering if there is a maximum luggage weight per passenger (assuming the passenger can pay the additional weight)?
There is no limit to the number of pieces of luggage you can check-in as a passenger; you just have to pay for whatever is above your included allowance. So the total weight of all your luggage is not important.
There is a limit to the weight and size of an individual piece of luggage but this is subject to whatever the airline decides.
If you were carrying something that (either by weight, or by dimensions) is not compatible with the existing luggage systems - the airline will accept the item as oversized luggage - for which there are other charges and restrictions.