My mom is visiting me, with a European passport and three months visa. Does she need to buy a one way ticket or a two way ticket? The travel agency told her she needs to buy a 2 way ticket.
According to http://uk.embassy.gov.au/lhlh/immi_travel.html
There is no requirement to have a return ticket when you arrive in Australia. However officers at the border need to be satisfied that visitors, working holiday maker and temporary residents are intending to and have the means to depart Australia before their visa expires. While not a mandatory requirement, you may wish to carry a copy of your bank statement to demonstrate you have sufficient funds to purchase an airline ticket to depart Australia.
So it doesn't appear to be mandatory.
However it is likely a good idea. Both because it may make it easier to convince the border guard you are a legitimate visitor and because on long haul journeys returns are usually significantly cheaper than two singles. Even if you end up paying a change fee the return may still come out ahead.