At many hotels, particularly those affiliated with major chains, it's perfectly fine. It's common for such hotels to leave a copy of your bill under your door sometime during your last night (they'll later add any charges you incur in the morning, of course) with a note that you can simply accept the charges to your card on file by leaving. There's usually a box by the front desk to drop off the keys so they can be reused. In some cases, there's an option to "check out" on the TV's interactive menu or the phone system, which lets them know you're done with the room. Many hotels now can simply email you a copy of the bill as well.
For example, this express check-out feature is expressly stated as a member benefit of Hilton's rewards program.
If your hotel doesn't offer an express checkout scheme, I would find it rude to leave without checking out, yes. They need to know when you're out of the room for housekeeping purposes. If you just disappear, the staff may take extra time to contact you with a copy of your bill. And while I've never had it happen to me, it's always possible some hotel manager will accuse you of staying past check-out time because you never checked out, and that's a bunch of trouble you don't want.
If you aren't sure about a particular hotel's policy, you can always ask when it's not busy. Even if the hotel doesn't have a formal express checkout policy, when I've known I'd want to leave in a hurry in the morning, I've asked to "settle the bill" the night before, and they've arranged things to expedite my departure.