Can be done but it's tight.
Hong Kong airport has a very good train connection to downtown. However, it's pretty far away: 45 minute ride, train goes every 10 minutes, so I'd budget an hour each way. Train ride is comfortable and scenic.
If you need two hours of travel, 1.5 hours for security and getting to the gate plus half an hour of buffer (you never know), that leaves you 1.5 hours in the city.
You could get off at Hong Kong Central, walk down to the pier and than hit the star ferry http://www.starferry.com.hk/en/service. It's super cheap and a great way to see the city. Get off at Tsim Sha Tsui and wander down the Avenue of the Stars along the waterfront. Time permitting (and depending on interest) you can also walk down Nathan Road which is the main shopping drag brimming with tailors and fake rolex sellers. To return, take a cab to Kowloon station or the Ferry back to Hong Kong Central.
Things are a lot easier if you get an Octopus Card with Airport Express tickets upfront. Octopus is a swipe-to-pay system that's accepted on public transport and many stores (including 7-11 which are abundant). https://www.travelchinaguide.com/cityguides/hongkong/airport-express-pass.htm