Welcome to the wonderful world of airline pricing!
Exactly how trips are priced will depend on a number of factors including the airline and the exact time you're flying, but as a generic answer...
Flights are normally prices on a per-leg basis, and then the price for each leg is combined to make up the total trip price. When you fly a round-trip, the prices for each leg will often be cheaper than if you just buy a one-way ticket. Open-jaw trips are normally priced at the same price (per leg) as round-trip trips, so normally you'd expect the overall prices to be similar for a round-trip as they are for a open-jaw.
So, when you buy NYC->ABC and DEF->NYC, it's fairly simple - it'll be the round-trip/open-jaw price for each of those legs combined.
However if you buy NYC->ABC, ABC->DEF, DEF->NYC there's a number of ways it could be priced.
The obvious way is that the prices for each of those three legs could be calculated and added together. This is normally referred to as a "circle fare", and obviously it is always going to be more than paying for the open-jaw as it's basically the open-jaw plus an extra flight.
However, it could also be priced as NYC-ABC, ABC-NYC with a stopover at DEF. Or as NYC-DEF with a stopover at ABC, and DEF-NYC.
Picking the first of these an as example (NYC-ABC, ABC-NYC with a stopover in DEF), you're now no longer purchasing a ticket to DEF, but a return trip NYC-ABC. If flights to ABC are cheaper than DEF - which they could be for any reason including more competition on that flight, a sale on at the moment, etc, then you're obviously going to pay less. Some airlines will charge you a little more for a stopover along the way whilst some will allow a single stopover for free - but even if you have to pay for it it's generally going to be cheap.
Most good travel booking websites will automatically look at the options available and work out the cheapest option for you. ie, they will compare the prices for the circle-fare option, the NYC-ABC return with a stopover, and the NYC-DEF return with a stopover options and then allow you to book the cheapest of the 3.
As an similar example, I recently had to book a ticket SFO-LHR which was going to cost around $1200. However I decided that I wanted to spend a few days in Istanbul on the way back, so I decided to try and book SFO-LHR-IST-SFO (with multiple days in both LHR and IST). You would expect that this would cost more, as clearly it's a much longer flight, however it actually cost only $1000 - cheaper that the direct option.
The reason was that it actually priced out as being SFO-IST, IST-SFO. ie, technically I bought a round-trip ticket to Istanbul, and just happened to have a stopover in London on the way. Airfares to Istanbul at the time were cheaper than airfares to London, so even after paying ~$100 extra to have a "stopover" in London, it still came out $200 cheaper!