I live in the UK. Photo booths propose different types of photos and it's often unclear which one is the most suitable for passport or - in my present case - Schengen Visas. Which photo booth option should I pick?
If in doubt on visa photos, by far and away your best bet is to find a real photographer
In my experience, they really weren't very much more - last time I had some unusual visa photo requirements I paid about 1.5 times the cost of a photo machine
You either need to go to a camera / photography shop, or a dedicated photography studio
In this case, you won't be dealing with an unhelpful machine, you'll be dealing with a person. You can talk with them about the requirements, and if needed they can look them up. They will understand the rules, and will take photographs to suit those rules. If needed, they'll retake them. After that, they'll print them at the right size.
Yes, it takes a bit longer than going to a passport photo machine. Yes, it costs a little more (1.5x in my experience) than a machine. However, you're dealing with a real human who does this a lot, who understands the rules. They'll help you get it right, and right first time.
The time and money you spend will be dwarfed by what you'd have to spend if the passport photo machine got it wrong! So, for simple normal stuff, machines are fine. For anything unusual/complicated, pay a tiny bit more, and have a friendly photographer sort it all for you in person!
If I am to spell it out in numbers, the Schengen visa photo is supposed to be 35 mm in width, 45 mm in height, and the length of the face should lie in between 32 mm and 36 mm. This is my reference: http://www.immihelp.com/visas/schengenvisa/sample_photos_germany.pdf
I went to a local pharmacy near me, with a print-out of the above document and the photographer there returned with photos in which the length of my face was less than 32 mm. Thankfully he was courteous enough that he redid the whole batch, this time complying with all the rules.
Coming back to your specific case, how much control do you have over the booth? I don't know much about photo-booths in the UK but if you are able to control it to get the results which satisfy the given specifications, then I guess the booth can work just as well as any other option. You can even cut the photo to the right size given that you get the size of your face correct (say in a larger-than-necessary photo). That's what my photographer did anyways.