I'm a student in Canada and this is first time I’m going to travel for long distance from Montreal to Hanoi (VN). I booked a ticket flight of Japan airline, Montreal-Chicago-Tokyo-Hanoi. I will transit at Chicago ORD. I have a visa visitor USA, it's possible to pass Security of airports?
Most US airports require that you pass security when you are transferring between international flights. Your visitor visa will enable you to do this, provided, of course, that is has not expired.
Because you have a visa, you should not apply for ESTA, which is only for those traveling under the visa waiver program (VWP).
However, as noted by blackbird57 in a comment, Montreal is one of several airports in the world where travelers clear US customs and immigration at the departure airport instead of on arrival. The transfer will therefore be handled in Chicago as a domestic-to-international transfer, and your bags ought to be checked through to Japan.
(I do not know whether you will need to clear customs with your bags in Tokyo, however.)