I'm travelling on the ESTA visa waiver, I'm now in the States but would like to go to Costa Rica for a bit (say three weeks) then come back to the States. All will be within my 90 day limit.
Is that allowed?
(I have my onward ticket home.)
Yes, no problem. You can exit and re-enter during the 90 day limit, and in fact if you leave North America (USA, Canada, Mexico) the 90 day period resets.
You just need to make sure your ESTA is valid for the entire period as well (they're usually valid for 2 years) and you have evidence of your travel plans and finances are in order.