The US Customs and Border Protection website has an answer:
How would my eligibility for a travel authorization via ESTA be affected if I was previously denied a visa, previously denied an immigration benefit, or previously committed an immigration-related violation?
If you were previously denied a visa, or previously refused entry to the United States, or previously removed from the U.S., your ESTA application will most likely be denied.
If you were allowed to board your carrier, you may be subject to additional processing upon arrival at U.S. ports of entry, and may be denied admission to the U.S. Applicants who are uncertain of whether they qualify for travel under the Visa Waiver Program (VWP) are encouraged to apply for ESTA authorization early, to allow time to apply for a visa, if needed.
Many people make the mistake of thinking that if they were denied a visa when they were a citizen of one country, then become a citizen of another country, they can travel to the U.S. under the visa waiver program using their new passport. That is wrong. The eligibility question about having been denied a visa does not ask whether or not the denial was made after you became a citizen of another country. It asks, "Have you ever been denied a visa?" - period. If you answer "No" then we determine that you have been denied a visa in the past, we consider that to be fraud, and you will be barred from entry for a number of years. It is better to admit the truth, and then apply for a visa, than go through the experience of being deported for fraud.
In some cases, you may not be sure how to answer this question because you were "allowed to withdraw" your application for entry, or a U.S. Consulate declined to issue you a visa because you did not have enough paperwork for them to make a decision on whether or not to grant you a visa. If this is your situation, we advise you to contact a U.S. Consulate and discuss what happened and ask them to tell you whether or not the withdrawal or declining to issue a visa constitutes an actual denial of entry or visa.
In other words the ESTA will likely be denied, but there is no harm in applying for one as long as you answer truthfully and fully about your previous denial. Lying or omitting your previous denial could lead to a deportation and entry ban from the United States. Also note that if rejected, you would not be refunded the $4 ESTA processing fee. The remainder of the fee is not charged for failed applications.