In theory, the International Certificate of Vaccination or Prophylaxis (ICVP) card serves as a record of all major vaccinations you've had, including things like Tetanus, Hepatitis, Rabies, Typhoid, TB, etc. Carrying one with you when you travel is a little like carrying a very abridged version of your medical records - specifically for vaccinations. The obvious thing this could save you is something like an extra tetanus shot given that you can show you only had one X years ago.
In practice, the ICVP really only serves a single purpose - to show that you have received a Yellow Fever vaccination within the past 10 years. The International Health Regulations from the World Health Organization (WHO) allows/recommends for countries to require proof of vaccination against Yellow Fever for all incoming passengers who have recently visited a country where Yellow fever is prevalent - even if only for transit. There are also a number of countries that will not allow entry at all without proof of vaccination.
If you have an ICVP there's no real need to travel with it at all times, but if there's any chance you will visit one of these countries on your travels then you should definitely make sure you have one, and have it with you when you travel!
If you don't have a ICVP, obtaining one is as easy as asking for it when getting a relevant vaccination. For Yellow Fever vaccinations, you will be given one without asking and the relevant area on the card should be stamped AND signed by the person certifying that you've been vaccinated. For most other vaccinations you will normally not be given one unless you ask for it, and they will normally not be stamped or even signed.