As far as I've ever found there's not something exactly like what you want, but there are a few workarounds.
Previously I used WorldMate, it's more of an upcoming trip tracker rather than a where I've been but it does show you some of where you've been. There's also TripIt which is similar. But they're both heavily skewed for planning and tracking future travel (and selling you cars, hotels and outings).
There are a couple of basic travel map sites, TravBuddy has one and there's WhereHaveIBeen.Info as well. But they're basically just sites to color a map, and generally at the country level. For examples here's the link to my TravBuddy map.
I'm actually surprised there's not a site out there that's more dedicated to this, maybe I've just never found it or maybe they always end up expanding into sales/reviews/etc.
BUT, there are a couple of ways you can get close to what you want yourself. First consider a photo sharing site like Flickr. Here you can tag photos with a location (your camera may already do it) and a time/date. Unfortunately I can't get into the site at the moment but there definitely was a map view and I believe you can control whether your photos are public or private. So, you'd need at least one picture per location, but that would do what you want. Kind of.
Another alternative is Google My Maps which (if you have a Google account) allows you to save personal anotations on Google maps. So you can drop markers where you've been, make notes, and add pictures/video. You can share the map, or save it for yourself. Here's the help pages which you can browse through to get an idea. This will not color in countries, and there's no real date tracking.
Of course, if you have an Android device with location turned on, there's also Location History (again that link is not working where I am but I'm pretty sure it's right, someone can edit it for me if it's not) which will track where you've been.
None of those cover everything you want to do, it would be possible to create a custom website that ties several of those together to get closer to a single travel tracking landing page, but I don't know of any public site that's done it.
A few people here have countries visited maps in their profile, so maybe there is a better solution out there.
EDIT: As David Richerby points out in the comments one drawback with most 'color the map' sites is that deciding what parts to separate is difficult. People disagree on what's a country, and borders and countries change over time giving problems to update the map. I guess the best way would be a system that took 'points on a map' data and combined it with some external (constantly updated) list of borders to produce the colorized map. Which actually doesn't seem too difficult if the data exists, it's the sort of mash-up that gets thrown together in hackathons all the time.