Do you get an I-94 when you drive in on a Nexus card? If you do, please check the I-94 database and then come back and let us know whether they've recorded your exit. I suspect that they will not have, because the I-94 database seems to be keyed on passport number.
In any event, it won't be a problem as long as you can prove that you left the country. A boarding pass for the flight to Europe ought to do that. Of course, you won't be able to show them a stamp in your passport since the EU border guards don't stamp EU passports (they might if you ask though).
IT's possible, but doesn't make much sense, to use the Dutch passport to enter the US. You would then have to worry about the VWP.
The more sensible option would be to check in to the flight with your Canadian passport, so the APIS information that the US authorities get can be linked to your I-94 record. (You are aware that the exit data comes from the airline, and not the border authorities of the next port of entry, aren't you?) It is entirely fine for you to check in to the flight with the Canadian passport and then show the Dutch passport when to the border guards when you arrive in the Schengen area.
I am a Dutch/US dual citizen. I have on many occasions flown from the US to the Netherlands checking in with my American passport and showing my Dutch one at the border. In the last couple of years, I've started checking in with my Dutch passport when leaving the US, to see whether there would be any problem with the absence of an entrance record for that passport. So far, there has not been. In these cases, though, I've not been flying to the Netherlands, only to the UK, Portugal, and non-EU final destinations with transfers in Germany.
So, to conclude:
- If you want to minimize the chance that you'll need to worry about proving you've left after the fact, fly on your Canadian passport.
- If you want to fly on your Dutch passport, retain proof that you've entered another country in case you are challenged the next time you enter the US.