You have to take your passport out of its wallet or cover when you give it to the border control agent. So there’s no issue of it not being able to be read. The range of the passports is relatively modest so they don’t “read” them until you step up to the booth.
Conversely, the border control agent NFC readers are designed only for near-range so they won’t pick up any other RFID or NFC tag unless it’s stuck on the passport itself. Which is another reason they ask you to remove the passports from their covers or wallets.
And as far as pre-boarding security such as the x-rays, metal detectors, millimeter wave scan, and so forth are concerned, the wallets and pouches are not a problem as long as you put them through the X-ray machine. They are simple foil or mesh faraday cages and the X-ray machine can easily see through them. If you keep them on your body, the millimeter scan (full-body human scanner) will detect them as an anomalous object and metal detectors may detect them as metallic objects and you’ll be pulled aside. If you’re lucky, you put your wallet in the little bin for X-ray scanning and you’ll go through again. If you’re unlucky, you’ll get physically examined, swabbed and (hopefully not) probed.
Related trivia: NEXUS cards have a much greater range and if you’re driving across the Canadian-USA border, border control ask you to remove them from their RFID covers well ahead of the border control agent, so they can read them while your car is still in queue. But you obviously aren’t driving to or from Canada so this doesn’t apply.