Are there public transportation options from Melbourne to 12 Apostles (and back) that would allow me to make a round-trip within 1 day (e.g. depart Melbourne at 7am and return to Melbourne at 10pm on the same day, spending a couple of hours near 12 Apostles and possibly nearby attractions). If so, what would be the price of such a trip? Or is booking a day tour the only option to get there without renting a car?
That won't work as a day trip.
You can get to the Twelve Apostles using public transport. However, it's one bus a day, so if you wanted to get off and sightsee, you'd need to make alternative arrangements for returning.
This link gives you details from Melbourne. It's train to Geelong, bus to Apollo Bay Great Ocean Road Visitor's Centre, and another bus to Warrnambool.
That gets you to Warrnambool railway station at 5pm.
The return trip might be first thing in the morning going initially to Waurn Ponds. You'd get back to Southern Cross at about 10:10 am the next day.
I'd be guessing the exact fare. https://www.ptv.vic.gov.au/tickets/fares/regional-fares/#paperticket
Contact V-line and speak to someone there to get more detail.
I'd recommend for tourism, it would be simpler just to book a coach tour that's tailored to the experience. For example