The most common option is to use the Flybus but there is also a competing company: Grayline. See if their schedule could suit you. Either way you are looking to spend €20+ per trip/person.
Keep in mind that a regular Flybus ticket takes you to the BSÍ terminal, which is at a certain distance from the city center. Since you will be arriving late and presumably are not familiar with the town you might want to consider the Flybus plus option: upon arrival at the BSÍ terminal passengers are dispatched to minibuses and driven to their hotels. I would suggest this option for a first-time visitor. Even if you don't mind walking and dragging your bags at night the weather may be slightly uninviting.
For the return trip, the choice is yours. You can meet up at the BSÍ terminal or choose the Flybus plus option again, and they will pick you up at your hotel if you want.
The trip from the airport to downtown Reykjavík takes about 45 minutes. Using a taxi is going to be expensive, like everything else in Iceland. It could make sense for a group of persons to pool expenses, but won't be much faster.
For the return trip consider Strætó line 55 (also departing from BSI terminal) if you will be leaving during the day. It's cheaper but the trip is a bit longer.
PS: if you plan to rent a car it makes sense to rent it in Keflavík and you bring it back there when you leave Iceland.
Also note that the vast majority of restaurants are closed at Xmas. Either stock up some food or be sure to book a table in advance and of course it won't be cheap. Welcome to Iceland!