There are a couple of possibilities.

 - [Eurolines][1] operates bus lines between several French cities
   
 - There are slower regional trains, the so-called [TER][2]. On longer
   trips it requires changing trains a couple of times. However, I am
   not sure if on these long trips the TER will be much cheaper than
   high-speed trains (TGV).
   
 - Have a look at [iDTGV][3], these are a kind of low-cost TGV. They are a bit cheaper than the regular TGV's. 
   
 - Train tickets are cheaper if they are booked in advance. The earlier
   you book, the higher will be the chance to find cheaper tickets. 
   
 - If you are flexible, avoid traveling during peak times (e.g. Friday
   and Sunday evening, early in the morning on the weekdays, ...)
 
 - You could also have a look at [rail passes][4]. It can be cheaper than buying the tickets for each trip separately, but there is no guarantee for that.   

 - There are overnight trains running within France. On longer trips that could be a possibility to save some money. Indeed, when traveling on night trains you don't have to pay for accommodation. 

 - If the objective is to minimse costs, you can also have a look at
   flights. [Easyjet][5] and [Ryanair][6] operate flights within France.


  [1]: http://www.eurolines.fr/
  [2]: http://www.ter-sncf.com/
  [3]: http://www.idtgv.com/
  [4]: http://www.seat61.com/Railpass-and-Eurail-pass-guide.htm
  [5]: http://www.easyjet.com/
  [6]: http://www.ryanair.com