Officially Cambodia's policy is possession of a "return ticket", which by definition means a ticket that takes you back from whence you came on the same type conveyance. A "return ticket" is the most common specification for proof of exiting the country within the time allowed used by countries all over the world.

However, most countries will accept an "onward ticket" in lieu of a "return ticket", as it shows your exit plans from that country.

Unfortunately most airlines check in policies mandate an air ticket as the only acceptable POOT (proof of onward travel).  This is due in part to the fact that most buses or trains go to the border, you then cross on foot and pick up a different bus on the other side, so your bus ticket doesn't actually take you across the border (even though your reservation might).  Whereas on a plane you clear immigration before embarking and can only disembark at the destination.

Also airlines have been in the game long enough to know that a $5 bus ticket to the border is a cheap way to try and get around the rules.  And while people still buy cheap airfare as throwaways, the number of people scamming with this routine is much smaller.

All that said, it is really up to the check-in counter agent to determine if you meet entry requirements.  Travelers have talked their way onto planes with only a bus ticket or a hotel booking.  But likewise travelers have been left behind at the airport for the same reason. Do you feel lucky?