Per this link, it looks like you need a koseki touhon (family register extract) showing that you're currently married to a Japanese citizen, which you can then use to apply for a visa:
Currently, all foreign nationals who wish to newly enter Japan need to apply for a visa except for those with re-entry permit [...]
(2) Spouse or child of a Japanese national/permanent resident:
Please obtain a Certificate of Eligibility or full copy of the family
register/certificate of residence/residence card and contact Japan’s
As far as I can tell a Certificate of Eligibility, spouse visa or any form of proof of residence in Japan is not necessary.
August 2022 update: Finally went through the process and got my visa. The koseki touhon was still required, and by far the most annoying part of the process. Aside from a copy of my spouse's passport, I did not need to provide any of the other documents listed in points 4-13 of @a20's answer: no invitations, bank statements, etc.
Note, though, that there are two separate application forms for
Short-term Stay Visa for Spouse of Japanese National for Visa Exemption Nationality (mm, visa for visa exemption!) and Short-term Stay Visa for Spouse of Japanese National for all Other Nationalities. I'm lucky enough to fall into the first group, but if you don't, you do need to provide bank statements etc.