Just to clarify first, travelling on rooftops happens on small train routes - local trains that stop at each and every small station, typically not run using electric engines, slow routes et al. For most long distance travel, this will not be the case. The stereotypical image of people hanging out of train coaches used as filler shots in montages in documentaries on India is usually that of the Mumbai Suburban Railway network (and boy is that crazy).
India is a massive country and airfares are nowhere near cheap as of yet for common people to travel with their family. Consequently, railways rather than planes or buses carry a majority of the inter-city traffic. Families plan trips months rather than weeks in advance and it's a crapshoot getting a reservation simply because there's so much demand to meet! Trying to book your tickets in advance is therefore very much advised.
Many trains have 'unreserved' coaches which you can buy (extremely cheap!) tickets for on the day of travel itself and get on-board. However, these usually packed full - and I mean full. Sitting / standing space only. Very uncomfortable if you're travelling long distance or overnight, as some journeys take 1-2 (or even three!) days to complete. Having said that, some of my most memorable train journeys have been when I've just gone to the station and gone in unreserved coaches.
Slightly costlier these are 'second sleeper' (two bunks per column), 'second sitting' (seats only), and 'three-tier sleeper' (three bunks in one column). These are reserved coaches and you technically, you need to have a reservation to get on - which means you need to book at least a week or a fortnight in advance. At least, the reality is you have to book even earlier sometimes. However, the implementation of how strictly rules are enforced really depends on what part of India you are in. Don't be surprised if a party with reservation for three climbs on-board with six people and they all share the space - it's part of how train travel works in India! And man, the amount of luggage they bring on (technically there are rules on how much luggage per person you can take on-board but this is never enforced). What I've see is in southern and western India, rules are implemented fairly strictly, i.e., you need to have a reservation to get on. In northern, central and eastern India, it is far more common to see people who have bought tickets but not gotten a reservation climb on board anyway and share seating space with strangers, or perhaps pay a bribe to the ticket checker to ignore the fact you don't have a reservation.
So even if you do have a reservation, you might end up having to share some space anyway. As a foreigner though you will find most people will go out of their way to be friendly with you, even give up whole bunks for you to sleep while they sit. Doing this for 'guests' is an essential belief many Indians hold. In any non air-conditioned or 'standard' coaches, you will be travelling with some of poorest in Indian society and you'll definitely be the centre of curiosity. Expect people to talk and be inquisitive about you, and don't be freaked out by it.
How easy it is to get a reservation is also a factor of how much you have to pay. It's easier to get reservations in air-conditioned coaches or express trains that take shorter times, simply because most people go for the cheapest trains. For you, the price difference will probably be inconsequential between second sleeper in the fastest train and second sleeper in the slowest mail train, so go for the express trains whenever you can.
Also, rlesko is right when he mentions foreign quotas. Seats on Indian trains are allocated according to quotas: there's a separate quote for senior citizens, women, armed forces personnel, 'general' travellers, railway employees...too many categories if you ask me! You can always buy general category tickets, but if you pay for the 'foreigner quota' ones then basically you're just competing with other travellers rather than 'everyone else'. Prices are higher though. On that note, you might also want to look at Indrail Pass.
An increasing amount of seats on any given train these days are reserved for 'tatkal' quota tickets (it means 'instant' in Hindi). Bookings for these tickets open one day before the day of travel and you pay slightly higher prices but if you're the sort of person who plans last minute, these tickets are your best shot. They still run out fast so rather than trying to go to the station a day before to book, get up early in the morning and book them online at IRCTC (the official railway ticket booking site) or Cleartrip.com / Makemytrip.com (two travel search sites that are authorised to sell tickets on behalf of IRCTC). You can show a printout of your ticket or even have them delivered as an SMS. You pay cheaper prices than the foreign quota tickets, and you get to make spontaneous plans. Win win!
