The updated visa waiver form asks for my parents' "family" name, as shown below. Does this mean my mothers birth family name or her married family name?
It's always the current name, since this is what she's legally known as. Any records - criminal, legal, etc that they might want to look up - or to check your back story, that's how they'd find her.
If they want the maiden name, they'll specifically ask for that.
If she is married to your father and took his family name then as @pnuts said it's an convention to write this family name if it is not the case then I can't help you sorry.
The guidance about Parents is not specific on this point:
Enter the names of your parents. These are required to complete the application. If you do not know the name of one or both of your parents, enter UNKNOWN, for each parent. This field can include the names of your biological, adoptive, step-parent or guardian.
In the Family Name field, enter your parent's family name. The family name is required to complete the application. If you do not know one or both of your parents, enter UNKNOWN.
In the First (Given) Name field, enter your parent's first (given) name. Do not include the middle name in this field. The first name is required to complete the application. If your parent has no first name, or if you do not know one or both of your parents, enter UNKNOWN.
There is though some guidance regarding 'family name' for the applicant:
The most relevant portion (emphasis added):
In the Family Name field, enter your family name as the family name appears on your passport under the Family Name or Surname field. The family name is required to complete the application.
This suggests that for your parents' names, you should enter their name as it appears on your passport, as well.
It is the name she goes by. If your mother has been known as Jane Smith for 50 years, the fact that she was born Jane Jones is not of interest to the visa people.
Of course, why her current name is of interest to them escapes me.
For info I just had the same issue today. I emailed ESTA Support in the USA and they confirmed that the mothers' Family Name is indeed her MARRIED NAME.
Hope this helps.