Traveling from Amsterdam to Rotterdam is certainly most convenient by train, most buses that do the run are mainly for people who need to do just a part of the travel and the buses are not much cheaper but they are much slower.
The Amsterdam to Rotterdam travel has three kinds of trains, the normal speed commuter trains and the high speed trains within the Netherlands and lastly the International high speed trains to Paris.
Tickets for the trains within the Netherlands are not timed and you can use any train on the day you have the ticket for. If you want to use the high speed trains you need to buy an extra 'ticket' for the surcharge. Again no reserved seats and no limitations on which train you take.
It is no use to get a fast international train for just the Dutch leg, the tickets are timed and very expensive. You might win a few minutes if you are at the station just before it leaves but as soon as you have to wait for the train you will lose all that advantage.
http://www.ns.nl/en/travellers/home will tell you all you need to know. If you travel only once, buying a ticket from one of the machines is the best option, bring coins as the machines do not accept paper money, or a card that works on it1.
Buying a paper ticket is €1 more expensive than using a chip card, but the chip cards cost €7.50 and you need €20 credit on it to use it on the trains.
Tickets are available online and you do not need to pay extra if you buy them that way, but payment options are limited, (but might include a way that works for you.) Your own home print will be as good as a machine bought ticket if you can buy online.
This is the new front of Rotterdam centraal station. Photo by Willeke, can be used under Creative Commons rules.
At this time the basic ticket Amsterdam-Rotterdam one way cost €16.10 (2019 price), reduction prices are irrelevant as you need to buy a year long pass to make use of those reductions.
Unless you can time your travel to make use of the fast trains, I would go for the normal trains and chose to travel outside the peak hours, so between 9 AM and 3 PM or after 7 PM, the trains in the peak hours are mostly very full but might have seats when you look at all carriages2 and mostly you will need not stand all the way.
The fast trains within the Netherlands, at this time called Intercity Direct, are faster and if you travel between the Centraal stations of both cities you are likely faster if you use them. If you need to change trains to get the fast train but can get a direct train from your starting station the fast train is often not worth it.
When the times fit your schedule, it might be worth the extra €2.40 for a single travel.
A point against the fast trains, they only have one stop in Amsterdam and just one in Rotterdam, so if you want to travel from one of the other stations in the city areas, you are not going to be able to take them efficiently.
Visa, master card, meastro, and V card are the accepted kinds of cards according to http://www.ns.nl/en/travellers/arrange-and-buy/tickets-and-passes/purchasing-tickets/ns-self-service-ticket-machine.html.
Note: Dutch railways distinguishes two classes: 1st class and 2nd class. A first class ticket costs € 26.57 single while a 2nd class costs € 16.10. With 1st class you can sit in first and second class compartments, with 2nd class you should sit in 2nd class compartments and should not travel in first class compartments at all, fines are likely if caught out. First class compartments are indicated by a '1' outside and inside, second class compartments are indicated by a '2'. First class is somewhat more luxurious ans spacious and has more often seats available especially during peak hours.