The rules are explained on the official EU website. It sometimes difficult to understand how they apply to a given situation but the page should be authoritative. In your case, my understanding is that you do need a visa to visit other European countries.
To understand why, you have to make a few distinctions. Assuming you are talking about a short stay, there is no such thing as a “EU visa” but several distinct area/countries with their own visa regime:
- Schengen area (most EU/EEA member states plus Switzerland)
- UK (not relevant for you as you already have the right to be there)
- Ireland (will not join Schengen to stay in the “common travel area” with the UK)
- Cyprus (blocked from joining Schengen as long as the state does not have effective control on the Northern part of the island)
- Romania (ready to join Schengen but blocked by other members' veto)
- Bulgaria (ready to join Schengen but blocked by other members' veto)
- Croatia (slated to join Schengen at a later date)
Croatia, Romania, and Bulgaria have rules modeled after the Schengen area and make it easy to visit for holders of multiple-entry Schengen visas or Schengen residence permits but are still separate. UK and Ireland have entirely separate policies and don't have any facilities for Schengen visa holders but do have to honor a few EU rules.
There are two ways in which being the spouse of an EU citizen can allow you travel to some or all of these countries without a visa but neither apply to you as far as I can tell:
- If you would reside in the Schengen area, you could go to Switzerland and all EU/EEA countries except the UK, Ireland and possibly Cyprus (with or without your spouse) but that's not your case as you reside in the UK.
- If your spouse would live somewhere else in the EU, you would qualify for an “EU family member’s residence card” in that country and could use it to go everywhere in the EU/EEA (including the UK and Ireland) and Switzerland with your spouse but that's not your case since your spouse lives in his or her country of origin and does not fall under EU rules about freedom of movement.
In all other cases (and in particular if you live in your spouse's country of origin), you have to follow national rules regarding spouse visa (i.e. you don't get an “EU family member” permit) and you do need a visa to travel to other EU countries. The good news is that as long as you travel with your spouse, this visa should in principle be issued quickly and free of charge.
Note than in the UK, British citizens have successfully argued that staying elsewhere in the EU for some time is enough to invoke EU rules instead of national rules (it's called the Surinder Singh route after the person who argued for it in front the EU Court of Justice). In that case, you would initially have applied for an EEA family permit instead of a family of settled person visa and you would fall under the second exemption I mentioned above.