From the Rail Planner app, I only see an option to reserve a seat for the Utrecht - Frankfurt train, is this correct? All of these trains are run by Deutsche Bahn, I think, which from experience does require seat reservations. Though I have also never needed to reserve seats before for trains to and from Vienna.
That's not quite right; the first train from Rotterdam to Utrecht is run by Nederlandse Spoorwegen. The next 2 are Deutsche Bahn. Your last train from Vienna to Budapest is interesting, it's a set of carriages owned by MÁV which are attached to the back of the Dacia Express overnight sleeper train [which is mostly run by CFR Călători] from Vienna to Bucharest in order to provide an evening train from Vienna to Budapest.
With that set aside, none of the trains require any reservation. In fact very few Deutsche Bahn trains require any reservation, even high speed ICE trains, it's just a few international routes -- your train between Utrecht and Frankfurt is not one that requires reservations.
That said though you may wish to make a seat reservation - I'd imagine many trains would be pretty busy on Boxing Day. For Deutsche Bahn you can make them at https://reiseauskunft.bahn.de/bin/query.exe/en?&rit=yes -- by going direct to train operating companies you can save the extra fees charged when you go through the Rail Planner app.
Also, I'm a bit worried about the transit in Frankfurt. If I miss the transit due to my train being delayed, do I need to reserve another seat again for the next train to Vienna? Also, that means I have to spend the night in Vienna, could I get compensation from DB for that?
I don't know if there are any specific extra German rules that would allow you to get compensation from Deutsche Bahn. Interrail passes are considered though a single ticket - and you are making an international journey so you would get some protection through CIV -- https://en.wikipedia.org/wiki/CIV_(rail_travel). The full regulations are at https://cit-rail.org/secure-media/files/documentation_de/passenger/civ/civ1999-f-d-e.pdf?cid=224515 and include under Article 32:
The carrier shall be liable to the
passenger for loss or damage
resulting from the fact that, by reason
of cancellation, the late running of a
train or a missed connection, his
journey cannot be continued the same
day, or that a continuation of the
journey the same day could not
reasonably be required because of
given circumstances. The damages
shall comprise the reasonable costs of
accommodation as well as the
reasonable costs occasioned by
having to notify persons expecting the
passenger.
I'll add if you read down that there are some exceptions to this -- and that exactly what is "reasonable" is left up to local rules & regulations. It can also be incredibly difficult (if sometimes impossible) to exercise these rights in practice.
It should also be noted that under CIV rules "carrier" does not mean the company that runs the train - article 3 defines it as:
“carrier” means the contractual carrier
with whom the passenger has
concluded the contract of carriage
pursuant to these Uniform Rules, or a
successive carrier who is liable on the
basis of this contract;
So basically you'd likely be claiming against wherever you bought the train ticket from, not Deutsche Bahn. Your ticket will have a code somewhere -- most (possibly all?) interrail passes will say "CIV 9901" -- like this: https://twitter.com/chantall40/status/957604653344182272 -- which means you'd need to claim against https://interrail.eu/en/ If you have a different code you can check it at https://uic.org/IMG/pdf/codification_4n_22112018.pdf (left hand most column)
I'll stress again that exercising CIV rights in practice can be very tricky, if you do think you'll have any issue with connections I'd strongly encourage you to speak to members of staff as soon as possible. I'll also add though that there are later trains between Vienna and Budapest, none of which require a reservation, so you can get a later one if needed as long as you are not delayed too much.
Interrail do have a form at https://www.interrail.eu/en/support/delay-compensation which allows you to claim some money back for longer delays. But I note in the terms for using this it says around hotels:
Claims for reimbursement of expenses incurred as a result of sustained delay, e.g. for alternative means of transportation, accommodation, etc., fall outside this policy. Such claims should be submitted directly to the railway undertaking where the delay was incurred.
My own view from reading those is that this condition on their compensation form is in violation of the CIV rights that should take precedence, but I'll defer to anyone else with more knowledge or experience.
Another thing, I remembered reading somewhere that I can use the interrail pass ONCE in a country to get where I want to start my journey, yes? That means I don't need to buy a ticket to get to Utrecht from Rotterdam?
The term for this is the inbound and outbound journey -- https://www.interrail.eu/en/interrail-passes/what-is-interrail/travel-your-own-country -- these are 2 days which you can use to upgrade an existing travel day so your pass can also be used in your home country. You can use as many trains in your home country as you need to and still only use up one of your inbound/outbound journeys. So yes you can use your outbound journey to cover both trains out of the Netherlands without needing to buy a separate ticket.
As a final bit of advice I'd strongly encourage you to confirm the times of those trains directly with the operating companies, I think that is always good advice but particularly when traveling on 26th December. The Rail Planner app is not always fully accurate -- and particularly with things like engineering works (which in some areas is more common at Christmas) and with operators that run a non-standard timetable it can sometimes provide incorrect information.
