I would say whether to bring it is a maybe. In the cities (at least in Costa Rica, I'm not familiar with Panama), there are often paved sidewalks, and if you're planning on walking longer distances, you may not want to carry your 3 year old when he/she gets tired of walking.
That said, sidewalks in Costa Rica are not generally built with ADA accessibility in mind and are not going to be as easy to navigate as streets in countries like the US. There may be random obstacles like stairs, steps, or drainage ditches in the middle of the sidewalk, as well as sidewalks that may be narrow at times, so expect to do some stroller-lifting from time to time, and be careful not to drive the stroller off the edge of the sidewalk, some sidewalks are as much as a foot high! In smaller towns, away from metro areas, you may expect to find varying condition of pavement, if it exists at all, so unless you have an all-terrain stroller with suspension and large wheels, you may find the ruts and bumps too jarring for your kids liking, and they may not want to ride in the stroller.
So overall, a stroller isn't ideal in those areas, but whether you'd want to bring one anyway depends on how much your older child follows directions like stay out of the road and whether you would be overly tired if you just relied on carriers and whatnot--the answer might be different for people with different personalities or kids.
If you already have the carrier(s) and find your children like riding in them, you may find it much easier to skip the stroller entirely and just rely on the carriers, and then you don't have to worry about taking them in and out of the stroller every time you need to climb stairs, or avoid obstacles. Even in the US where there usually are good sidewalks, the carrier affords a lot more flexibility about where you can go!