As I understand it, your brother stayed for 2 years after he stopped studying? He can apply for a visa, but his chances of getting it might not be very good.
People on F1 are almost always admitted on "D/S" on their I-94, which means they do not automatically start accruing "unlawful presence" on any date. Assuming he didn't apply to USCIS for some benefit and was denied for being out of status, and he didn't receive a final order of deportation, he didn't accrue any "unlawful presence" before he left, and he does not have a 3-year/10-year unlawful presence ban.
But just because he doesn't have a ban doesn't mean he will get a visa. Visa officers have wide discretion to deny most types of nonimmigrant visas, including visitor visas, for "failure to overcome the presumption of immigrant intent" if there is anything they don't like about the person's case. And a past history of overstay is definitely a negative factor.