TL;DR: Yes, as a non-EEA citizen, you need to carry a passport in the situation you describe in your question.
First of all, both Austrian and German national law require all persons to carry a passport or other recognized travel document when crossing their respective borders. This requirement also applies to the crossing of Schengen internal borders, although there are no permanent immigration checks conducted. At least in theory, there should be no permanent immigration checks at the German-Austrian border. In reality, since 2015, immigration checks when travelling from Austria to Germany are quite common. German police operate permanent checkpoints now at all motorway crossings between Austria and Bavaria and trains from Austria to Germany are very often inspected at the first stop in Germany. From my own experience the last few years, I would estimate that at least 2/3 of all trains from Salzburg to Germany are inspected by the federal police in Freilassing. The police walks through the train and very obviously ignores all white passengers, while asking passengers with a darker skin colour for id. I've always been 'waved through' when driving by car from Austria to Germany and never stopped, but then again, I am a white man driving a car with German license plates.
Then we have the special situation with Austrian trains between Salzburg and Kufstein, which drive through Germany without stop. Due to the mountains and geography, it is for all practical purposes easier to connect the westernmost parts of Austria with the rest of the country by driving through Germany. These trains have, since 1957, been operated as so called privileged transit traffic, meaning that the trains are not subject to border controls. I read in the English Wikipedia article, that this arrangement ceased in 1997 when the Schengen Agreement went into effect, but I believe that is wrong. I can at least find pictures of train destination plates from 2005, where the trains were declared as so called 'corridor trains'. The agreement surely had more practical impact before the Schengen agreement, as it meant, that there was no passport control on these trains, although crossing the Austrian/German border at that time usually ment having your passport or id checked. Now, it still has relevance for customs. Even though both Austria and Germany are in the EU, some consumer goods, e.g. tobacco and coffee, are significantly cheaper in Austria and can only be brought freely into Germany in limited amounts. It also means that the requirements for crossing the border I mentioned in the introduction (carrying a recognized travel document) does not apply on these trains.
There is however one further 'when' and 'but' to this story, since you are asking about the requirements for a non-EU (I assume you actually mean non-EEA) citizen in this situation. You are as a non-EEA citizen in Austria required by law, upon request, to be able to show your passport to the authorities without unreasonable delay, which in law is defined as 'within one hour'. So at last, even if the fact that the train drives through Germany does not in itself trigger a requirement to carry a passport, you would need to carry the passport anyway, as long as you are more than one hour away from where you actually keep your passport.