In summer last (2012) year I visited Georgia, and I was with my friends in David Gareja. They were tired, and I decided to climb a hill south of the monastery, on the top of which is a small chapel. A bit below, I found a rocky monastery:
After I returned, they said the taxi driver was angry because he had to wait so long and he said I had crossed Azerbaijan border, and there's a border conflict and I could get shot.
However, there were leaves with prayers in Georgian, left by Georgian pilgrims (at least I think they were prayers, because I can't read Georgian):
So I don't think that entering there is restricted. I didn't see any soldiers there, and if it was really such a conflict area, I would expect at least a few soldiers observing the area, as well as border markers.
So what does the situation actually look like? Is this hill really a conflict area? Is it really prohibited to access the monastery? Or are the pilgrims' and tourists' visits to the area allowed/tolerated? And the taxi driver was only saying such stories because he would like the tourists to see the monastery below and leave after 20 minutes, so he can take another group the same day?