You can use American Airlines AAdvantage miles to buy tickets on a British Airways flight through American Airlines. The flight has to be listed as an available flight for miles redemption. Not all flights are eligible for purchase by redeeming miles regardless of if the flight is on AA or a partner airline. If the flight is eligible, it still must be booked through AA regardless of which airline is operating the flight. 

I have flown on three different airlines for the same journey (Envoy, British Airways, Cathay Pacific), all booked through AA on the same booking and ticket. Calling a partner airline directly will not work. They can not use nor convert your AA miles.

This all can be done through the AA website, mobile app (may not be useable for some international travel), gate agent, or live phone operator (additional fee/cost may apply). It may be worth it to use a live operator to find the flight that you want. 

In my experience, live operators have been able to book miles tickets that were not visible on the app or website. This has been especially true on multi-leg journeys when I wanted a specific routing, time, or operating airline.