When you go to the I-94 website, you have to agree to a warning:
By accessing this website, you understand and acknowledge that:
You are declaring under penalty of perjury pursuant to 28 U.S. Code §
1746 that you: (1) are only seeking records about yourself, (2) are
seeking records about someone for whom you are the legal guardian, or
(3) you have the consent of the person whose records you are seeking.
You are not authorized to access this website to retrieve records of
another person unless you are the person's legal guardian or you have
the person's consent.
Unauthorized or improper use or access of this website, including the unauthorized or improper modification, destruction, or disclosure
of any information or data contained herein, is expressly prohibited,
and may result in civil and criminal penalties.
The access and use of this website is subject to monitoring by DHS for administrative, law enforcement, or criminal investigative
purposes, inquiries into alleged wrongdoing or misuse, and to ensure
proper performance of applicable security features and procedures. DHS
may monitor the access or use of this website without further notice.
You may not process classified national security information on this
So at least on paper, it violates US law for someone else to look up your information unless they have your consent. But practically, there's not a deeply strong mechanism to enforce that, and people or entities outside of the US might not care that they're breaking US law. I wouldn't be surprised if there was some monitoring in place to detect bulk queries, brute force attacks, and abuse of the system, but it seems unlikely a rogue employer or someone else performing a single lookup would be caught and face consequences for that even if it is illegal.
If you click through to the privacy notice for the website, it does say that "CBP will retain the information submitted when attempting to access records through this website for 3 months for audit and system performance purposes," so there's at least a theoretical possibility they could identify some types of misuse from those logs. If you think your records have been misused, you could conceivably try requesting a copy of those logs that pertain to you; I have no idea if they'll actually give them to you (or refuse because you're not a citizen/permanent resident, or perhaps misunderstand your request and just send you your I-94) or do so in a timely fashion or whether they'd be of any use.
So yes, anyone with the required information from your passport can see your travel history (or at least the potentially incomplete version displayed by the website).