That's a lovely plan! Michael Hampton is partialy correct, in that the 5pm time refers to the café/buildings in the park and not the footpaths. However, the gates to the Gardens close around dusk each night. If you'll be going before March, you will have until 8pm to enjoy the Gardens. For a full schedule, see the hours shown HERE.
Additionally, I would recommend that rather than walk through the Gardens after the chair, you take the wonderfully scenic walk along the waterfront. It will take you right past the Opera House and provide lovely views of the harbour, the gardens, and the bridge. Google's route may be slightly quicker (no more than a few minutes), but it is much less scenic. If you want to walk through the gardens, I recommend doing so during daylight. It was always one of my favorite places to spend an afternoon in Sydney. Have fun!