I have an expired F-1 visa and am travelling back to Nigeria from the US to renew it, with a layover at Amsterdam (AMS). The [airline] counter in US says I can't layover in AMS with my expired visa.
Is this confirmed or did she misunderstand me?
TIMATIC, the database used by the airlines, states:
Visa required, except for Nationals of Nigeria, holding onward tickets transiting by the same or first connecting aircraft and meeting one of the following conditions:
- holding a valid visa issued by Bulgaria, Canada, Croatia, Cyprus, Ireland (Rep.), Japan, Romania, USA or United Kingdom, traveling to any non-Schengen Member State;
- returning from Bulgaria, Canada, Croatia, Cyprus, Ireland (Rep.), Japan, Romania, USA or United Kingdom after having used the visa;
Note it says "one of the following conditions".
Because you're returning from the US after having used the visa, the visa does not need to be valid.
In other words, the airline counter was wrong, and since they're obviously too lazy to look up the rules, I strongly recommend you to print the linked page (click on the word "Timatic" and the page will show)
You do not need an airport transit visa for Amsterdam.
According to the The Embassy of Netherlands in Nigeria,
You do not require a Schengen airport transit visa, if you:
- Hold a valid visa or residence permit issued by a Schengen state.
- Hold a visa valid for a member state of the European Union or the European Economic Area, Canada, Japan or the United States.
- Hold a valid residence permit issued by a member state of the European Union or the European Economic Area.
- Hold a residence permit issued by the Principality of Andorra, Canada, Japan, the Republic of San Marino, or the United States, that guarantees unrestricted right of return.
- Are a family member of a citizen of the European Union, the European Economic Area or Switzerland.
- Hold a diplomatic passport.
- Are on-duty aircrew and a national of a country that is a party to the Chicago Convention on International Civil Aviation.
The point in bold is important in your case (emphasis is mine). As your F-1 visa has expired, the USA would not let you in using this visa, that is you do not have an "unrestricted right of return" to the USA.
Therefore, unless at least one other point applies, you do need an airport transit visa to transfer in AMS. Do note that whenever a visa to any country is mentioned, it is always meant to be a valid, non-expired, non-cancelled and, in case of a single-entry, unused visa.
The site as linked above also contains information on how to apply for the transit visa.