US law says you must leave and enter the US with your US passport, but the US border officers do not actually look at the documents of people who are leaving, so they do not actually enforce that part of the rule.
As for returning, as others have noted, you can get into the country with your naturalization certificate, though you may have to do more explaining than you would have to if you had a passport. I think you could in theory be liable for a fee to gain exemption from the requirement, but I have read on this site that they do not actually impose one.
The biggest problem with returning to the US on the strength of your naturalization certificate would most likely be the airline. They might not accept your naturalization certificate as proof of admissibility in the US, since it doesn't comply with travel document standards, and if you lack a US visa in your Hong Kong passport, they might refuse to let you board.
Therefore, getting your US passport in Hong Kong seems like the safest way to go.