Entering and leaving a country with different passports is indeed not advised. If the country uses computerized records (like the US), the entry and exit could conceivably fail to be matched and you risk being marked as an overstayer. If the country rely on stamps and checks passports on exit (like the Schengen area), border guards will demand to see the passport you used to enter.
But that's not necessarily what your friend did. For example, he could have done this:
- Enter and leave the US on the Venezuelan passport
- Enter and leave the Schengen area on the Costa Rican passport
- Enter and leave the US on the Costa Rican passport
In all cases, he would have entered and leaved each country with the same passport. Two things are important here:
- It's perfectly possible to leave one country on one passport and enter the next one on another. When flying to Europe, you need to show the passport you used to enter the US to the airline (which will communicate its details to the US authorities) and you can show whatever you want to the Schengen border guards. When flying to the US, you would need to show the new passport to the airline (for the same reason as before: the US require airlines to communicate the details of the passport you used on all legs of your trip to the US, i.e. not necessarily only the last flight) and the old one to the Schengen border guards (they will want to see the entry stamp). Upon landing, you would use the new passport again.
- It's also generally possible to enter using a valid passport and a valid visa in another passport (but see the first paragraph of this answer). Typically, the visa would be in an invalid/expired passport from the same country and having two citizenships is a little more exotic but there is no reason it should be impossible.
Since Venezuelan and Costa Rican citizens can all enter the Schengen area without visa, it does not really matter and your friend probably used his Costa Rican passport on the European side (especially considering the fact that in the Schengen area you need a passport valid for some time after the planned departure date). What would be tricky is if your friend was, e.g., a citizen from Venezuela and Columbia (don't know if that's actually possible). Since Columbian citizens need a visa to visit the Schengen area, he would really need to use the Venezuelan passport. But since that passport was expiring, he shouldn't be able to enter.
Alternatively, he might just have been lucky.
Note: Travelling with two different passports explains all this very clearly.