I'm not sure wether the Dazaifu temple is considered to be inside the city limits and accessible with the tourist pass.
There are two versions of the Fukuoka Tourist City Pass. From the official website:
- Fukuoka City（not applicable to Nishitetsu Train）: Adult 820 yen, Child 410 yen
- Fukuoka City and Dazaifu: Adult 1,340 yen, Child 670 yen
So the more expensive one will include the train to Dazaifu. This is also noted in the brochure with more details on what trains you can use: