Yes, it can get crowded! Buy your tickets in advance! Or it might be an hour and a half before you get on a ferry. [Note I base this answer on our experience on one particular Sunday, but it is reasonable to suppose that the situation is similar, at least on weekends, most of the year.]
There are frequent departures (every 15 min), at least in the morning and late afternoon/evening. We thus assumed there'd be no problem. We got to the terminal at around 10am (Aug 24th, Sunday), but unfortunately the next ferry with seats was 11:30am.
One reason for this lack of seats was the scalpers who had bought a bunch of tickets, for precisely this purpose. They were openly and blatantly selling the tickets for the 10:30am ferry (for 30 HKD or about US$4 more than the original price of HK$172 for the weekend trip), right in front of the ticket counters.
Being desperate to get over to Macau earlier rather than later, we bought the tickets from these scalpers. This I suppose is how free market Hong Kong works!