Obviously your question is a bit old and at best you will have some fond memories of what used to be my home at the time you wrote your questions. But I hope that others with similar questions may find my answer helpful.
How likely is it to see the aurora in Iceland in late December? Does it involve getting some organized tour, or just stepping out of the house?
The issue isn't so much the aurora (IMO aurora forecast website) which depends entirely on the sun activity, but rather the cloud cover. From experience this varied a lot during my eight years in Iceland. I would not feel comfortable to make a claim one way or the other. In the end it is a matter of luck. No sugar-coating from me. However, staying a bit longer increases your chances. It may also be possible to catch a better view near body of water inland, such as Þingvallavatn and generally avoiding the light pollution increases your chances as well, provided you have a clear sky.
Organized bus tours are available. The way this works is by purchasing a ticket and hoping for the best. At the time I lived there and hosted couchsurfers, the way it worked was simply to reuse the same ticket next night (provided you don't have other plans) until you finally get to see the aurora (or have to leave). I know for a fact that some of my guests took three tours until they were lucky to see the aurora, but paying just once.
Do we need a 4x4 car in the winter?
Depends where you plan to go. If it's just the roads to the main tourist attractions you're typically fine with a "normal" car. However, there are a few exceptions, such as the mountain pass near Hellisheiði on the way from Reykjavík to Selfoss (or even just Hveragerði). That mountain pass is regularly impassable during winter. Check for current warnings with SafeTravel.is on a regular basis! I have a friend who lives in Selfoss and commutes to Reykjavík (actually Hafnarfjörður, a satellite town bordering the capital) nearly every work day and he doesn't use a 4x4. However, if you have more daring plans a 4x4 drive alone may not even be enough. In such a case I'd always recommend to stick with a local guide!
As a side note: spiked tires are mandatory during the winter season. This definitely applies to December, but I don't know off the top of my head when it starts and ends (March or April?). A local would know. In any case you can expect your rental car to have such tires, no matter if it boasts a 4x4 drive or not.
Sites I visited and enjoyed during winter include: Gulfoss, Geysir/Strokkur, Þingvellir (very picturesque during any season, even without the aurora), Borgarnes/Hvanneyri, Hveragerði, Selfoss and Hraunfossar/Barnafoss.
What other attractions are accessible and worth visiting during the winter?
I can attest to the fireworks craze Kris described in his answer. In my first winter in Iceland I lived near the shoreline in Reykjavík and that's where the official fireworks were displayed. The subsequent years were no less crazy, but I had moved a few hundred meters inland by then (no direct view of the shoreline).
The attraction (out of the list from the previous bullet point) I enjoyed most was Hraunfossar (translates to "lava waterfalls") and Barnafoss ("children's waterfall", make sure to read the story on the nearby signs). Both are in close vicinity of each other. It's a whole range of smaller waterfalls coming out of a vast lava field and it's quite spectacular both in summer and winter.
But if you are also interested in culture, go to the nearest outdoor (or indoor) swimming pool. No worries, they're all heated. And if you're lucky it'll snow. It's so exhilarating to take a swim and whenever you surface cool snowflakes land and melt on your back. Enjoying a hot tub (preferably in a public swimming pool as well) is also great while it snows and you get a chance to chat with locals. All public swimming pools I visited had at least one hot tub! Of course a cabin and hot tub next to Þingvallavatn without the light pollution is probably even better as you may be able to combine a communal experience with cozy warm water and aurora.
If you prefer a bit more touristy alternative to the public swimming pools, though, you could plan for a stop in Laugarvatn (the place is called Fontana) on your way from Reykjavík to Gullfoss/Geysir, which is a fancier version of a public swimming pool with a nice view on the neighboring lake.
Finally, and I kid you not (!), take the bus from your hotel in Reykavík to Elliðaárdalur, a picturesque valley (-dalur) in the city. It's also great any time of the year.
If you bring cushions (to sit on) the hot tub by the shoreline near the light house in Seltjarnanes is also a great place to relax. In summer you can skip the cushions and bring wine and time instead. Great sunsets are to be enjoyed from that place (mostly in summer, though).
Is Iceland a popular destination over Christmas? I.e. is accommodation and car rental going to be hard to find and/or expensive?
Main season is between May and September (or early October), if I remember correctly. So you're visiting off-season. But Iceland is generally expensive. Having been a couchsurfing host in Iceland that may be one way to save on expenses (but I disliked the one guest who merely used me for free accommodation, i.e. as "hotel replacement"). They (the locals, no matter if foreigners or Icelanders) will also know where to shop for groceries and can show you around. If you're lucky they'll have a car and you can hitch a ride.
The note about holidays is kind of a biggie, as Kris pointed out. Several of my couchsurfers got bitten by that. Especially in the capital area you needn't worry, though. There will always be some open restaurant willing to feed you. It's just more expensive than buying groceries in a discounter. This depends on the location though. Smaller towns (and that's pretty much anything other than Reykjavík) will sometimes not offer that luxury. But if Reykjavík was your base camp in 2012 I'll hold that you would have been fine even back then. And I know from the statistics that the number of tourists has been growing since then. So the situation will have improved.
NB: You may want to get something to strap under your shoes while visiting Iceland in winter. I've tried plenty of different solutions and the best I've found were called "YakTrax pro". The only time you have to be careful with those is when you get from the wet street into a shop with tiled floor or similar. Because these things don't provide grip on surfaces such as tiles. But they work excellently on snow, ice and even plain ground.
One more thing. If you have seen photos of the aurora on Instagram or some such, don't be fooled. Oftentimes these photos were taken with prolonged exposure, which catches even the faintest Northern Lights and makes them look spectacular. With the naked eye it often doesn't look quite as spectacular.
And an extra warning is probably also in order: Icelandic nature can be treacherous. It is very scenic and beautiful, but especially off-season you should rather spend some bucks on a local to guide you (or even tell you not to take a particular trip) than end up with the folks from Bjorgunarsveit (rescue services) or frozen to death on top of a mountain because that sudden cloud made it kinda hard to find your way back down. So trust the locals and their judgment.
