It is possible to reach Ilha Grande starting from Rio after 2 PM. How late depends if it is a Friday or not.
There are 3 ports to Ilha Grande: MANGARATIBA, CONCEIÇÃO JACAREÍ and ANGRA DOS REIS, in order from closest to furthest from Rio. Boats regularly depart each of these ports. On all days other than Friday, the last departure from CONCEIÇÃO JACAREÍ is either at 6 PM or 6:15 PM, depending on which site is correct. Friday, there is a departure at 9 PM according to both http://www.ilhagrande.org and http://www.caicarailhagrande.com.br.
ANGRA DOS REIS is risky as the last departure is 3:30 PM or 4 PM, depending on which schedule is accurate. MANGARATIBA has regular departues only until 2 PM which is too early, except on Friday when a 10 PM departure is scheduled.
To each CONCEIÇÃO JACAREÍ from Rio's GIG Airport it takes 1h32m according to Google Maps. It might be longer depending on traffic and being in an unfamiliar place, so I gave myself 2h30m just to be safe. Therefore one can expect to arrive on time to depart for Ilha Grande any weekday by leaving Rio before 3:30 PM which is the actual answer to my question. On Fridays, you can depart up to 6:30 PM instead.
Synchronizing this with a bus schedule is pretty difficult as one has to get to the bus station, wait for a scheduled departure, etc. For this reason, using a rental car to reach Ilha Grande late in the day is the best option. Cost is surprisingly low: $90 for 3 days of car rental, $30 for the boat between Ilha Grande and CONCEIÇÃO JACAREÍ (both ways), $24 for 3 days of parking. Total $144 CDN or $109 USD (at today's exchange rate). If you drive manual instead of automatic, the rental car goes down to $68.