Does 'mother' on the ESTA application have to be my birth mother as it is on my birth certificate, or can it be the mother who brought me up?
'Mother' is generally defined as your biological birth mother - unless you were legally adopted; then it is your adopted mother.
If your aunt or grandma or neighbor brought you up, no matter how great your relationship or how laudable their work might be, this is of no legal relevance.
Note that it is perfectly acceptable to write 'unknown' if your biological parents are unknown to you. I have seen that successfully done, in ESTA application, and also in naturalization.
Your ESTA application should match your documents.
If only the name of your birth mother appears on any of your documents, that is the name you need.
But if you have adoption papers or other legal documents that transfered parential powers to the person who brought you up, the name mentioned in those papers can be used.
In case of only an informal exchange of guardianship and you not happy with using your birth mothers name, 'unknown' or 'NA' might be accepted but I do not have the knowledge to be sure about that.