Do I need to fill up a separate application for BIVS endorsement or is
it handed out automatically?
There isn't a special form/checkbox to fill, but you must apply at a UK/Irish visa application center in India or China. Applications cannot be made at centers outside these two countries during the first phase of the scheme.
You will know your visa falls under the BIVS scheme if it carries a 'BIVS' endorsement.
Also, If I enter the UK, and then travel to Ireland, can I re-enter the UK again?
If you have a British visitor visa and want to also visit Ireland, you must travel to the UK first (and vice-versa). After entering the Common Travel Area you are then free to travel freely between the two countries until your leave to enter (as shown on the passport stamp) expires.
You may also enter the UK first on an Irish visa in order to transit to Ireland. The opposite direction (entering the CTA through Ireland on a UK visa) is not allowed.
Sources: UK Government's official BIVS page and the Irish Government's official BIVS explanation document.