For your British passport, the controlling reference is Paragraph 12 of the rules, which says in part...
A person claiming to be a British citizen must prove that he has the
right of abode in the United Kingdom by producing either:
- a United Kingdom passport describing him as a British citizen or as a citizen of the United Kingdom and Colonies having the right of
abode in the United Kingdom;
There is no requirement for the British passport to be current. The operative words are "
describing him as a British citizen" so the photo must still be a reasonable likeness and the personal details need to be consistent. If you expect difficulties with an IO, then carry a print-out of the rules and show it to him.
For the South African passport, the answer is no, it cannot be used alone. It needs to contain a valid and current entry clearance. To get an entry clearance you need to apply for it. They may reject the application (i.e., send it back to you unprocessed) if they learn you have a legitimate claim to citizenship. I don't know if they keep the fee or not in those cases.
Longer term, it may be cheaper to simply renew your passport. I just renewed mine without any additional expedite features and the turn-around time was 12 working days.
To help make this answer more generic, and to satisfy those with a penchant for absolute detail... Passports issued prior to the 1981 Act are not included in the rules. British passports issued between 1982 and 1998 can be problematic. Also old style passports that identify the holder as a "British Subject" or "British Overseas Citizen" or any other variant of 'britishness' that has occurred since passage of the 1914 Act will need to check with a lawyer first.
Also for completeness' sake, any sort of British visa in a foreign passport remains valid even if the foreign passport has expired. This holds for anything from entry clearance (which have not expired) up to and including the Right of Abode sticker.