I couldn't find any information about denying entry for transit visa holders (whether official, semi-official, anecdotes, etc.) Many sources indicate that it's quite rare to be denied entry to the US. According to this official summary (as of March, 2016) Out of more than 1 million visitors/day, only 367 visitors are denied entry and deemed inadmissible, that's like 0.03%. There's no mention of specific visa types.
Anyway, continuing my research about this, I found the term that CBP use in such cases, it's called "Expedited Removal". The expedited removal is basically what happens to a people who are denied entry at an airport or any point of entry. All official documents mention "removal" of the person from the US, none of them mentions anything about where to. Hence, I think it's safe to assume that you will be escorted to your next flight in your itinerary, as long as it's "leaving" the US soil.
Most of these expedited removals apply to people who are found to be breaking visa rules, that's quite hard to apply for people transiting through the US. The anecdotes speak of people who are found to be making money in the US (eg. working illegally while on B visa).
Also, the CBP officer may offer you another option beside "Expedited Removal", which is withdrawing your admission request, and then choosing to leave the US voluntarily. This will have less legal consequences, especially when it comes to the period in which you will be prohibited from entering the US, which is 5 years in case of expedite removals.