There are a couple of possibilities.
Eurolines operates bus lines between several French cities
There are slower regional trains, the so-called TER. On longer trips it requires changing trains a couple of times. However, I am not sure if on these long trips the TER will be much cheaper than high-speed trains (TGV).
Have a look at iDTGV, these are a kind of low-cost TGV. They are a bit cheaper than the regular TGV's.
Train tickets are cheaper if they are booked in advance. The earlier you book, the higher will be the chance to find cheaper tickets.
If you are flexible, avoid traveling during peak times (e.g. Friday and Sunday evening, early in the morning on the weekdays, ...)
You could also have a look at rail passes. It can be cheaper than buying the tickets for each trip separately, but there is no guarantee for that.
There are overnight trains running within France. On longer trips that could be a possibility to save some money. Indeed, when traveling on night trains you don't have to pay for accommodation.
If the objective is to minimse costs, you can also have a look at flights. Easyjet and Ryanair operate flights within France.
