The answer depends very much on your degree of risk aversion, and on how much of your "like to be early" buffer you are willing to lose to traffic or other delays en route.
As Michael Hampton noted in a comment, the drive is almost entirely motorway, so depending on your driving habits and the traffic, you might be able to average 75 miles an hour (120 km/h) or you might not do better than 45 (72 km/h). So I'd probably allow six hours of driving time, plus time for breaks to refuel and whatnot.
If you are willing to arrive at 5:30 if the traffic is heavy, you could probably get away with leaving at 11:30. But if you are not, and being there at 4 is truly important to you, you should leave at least by 10, and if you are strongly averse to risk and/or the stress of worrying whether you'll make it, you might want to leave closer to 9.
Traffic on I-95 in Connecticut can be truly horrendous. It once took me something like 6 hours just to drive from Manhattan to New Haven, about 75 miles (120 km), but that was the Sunday after Thanksgiving and there was a particularly nasty accident on I-95. I was also on the road once immediately after a bridge collapsed, but I was not driving and I did not note the precise duration of the trip.
Horrific anomalies aside, I would leave more time if it is Sunday or Friday; three hours from New Haven to New York wouldn't be terribly surprising.
You might give yourself some flexibility by planning to take a little detour for a nice lunch or something like that, somewhere closer to the airport. If the traffic turns out to be bad, you can cut that short, or skip it altogether and go straight to the airport.