It's do-able, but you won't get a lot of time in the city.
Unless you feel like paying for a taxi, your best option to get to the city is BART which leaves directly from the Airport. Presuming you're there on a weekday, BART leaves every 15 minutes (20-25 minutes evenings and weekends), and takes about 30 mins to the city. Allowing time to get from your plane to BART, buy a ticket, etc, you're going to be talking pretty close to 1 hour.
On return you're going to need to go via security which at SFO can take anywhere between 5 minutes and close to an hour, depending on your status/airline/terminal/time of day. Being an international flight, you'll probably start boarding around 45 minutes before departure time. So allowing for travel time, security, boarding and contingency, you're going to want to be leaving the city at least 2 hours before your flight time.
So out of your 5.5 hours, we've used up 3+ hours, giving you a little over 2 hours on the ground. Whilst that's certainly enough to see something, it's not much. It should be enough time for a quick wander around Union Square (BART to Powell st), Chinatown (BART to Montgomery) or The Ferry Wharf/Embarcadero (BART to Embarcadero) - but it's not enough time to safely get to Fisherman's Wharf or most of the rest of the city - at least not without catching a taxi.
BART will cost $8.25 each way. There's no return tickets as such - when you buy your ticket just make sure you put enough on it for the return journey to avoid having to buy a ticket when coming back.