Can I apply for NEXUS if my green card is expired but my case (through marriage) is pending and I have an extension that allows me to work and travel in/out the USA until the end of 2018?
According to this US government website, you need to be a lawful permanent resident of Canada or the U.S. (or a citizen of either country; or of Mexico, if you participate in Mexico's trusted traveler program), so your eligibility for NEXUS is going to hinge on whether, at this time, you are a permanent resident or not. My understanding is that it is merely your evidence of permanent residence that has expired, not your permanent resident status itself.
If indeed you are legally a permanent resident now, apply. If not, wait until you are.
In case you aren't a permanent resident, if you happen to be a Mexican citizen and want to fast-track this, join the Mexican trusted traveler program and then you will qualify for NEXUS irrespective of your residency status in the U.S.