Yes, you need to be concerned. If you buy this then you could become responsible if the EPA or state DEQ requires remediation due to ever-increasing regulatory oversight and lowered thresholds for soil / water contamination,.. Also if this property has water wells nearby and people start getting sick there is liability associated with that if linked back to these contaminants.,
I have had a dirty Phase 1 ESA come back too when contracting a former gas station parcel. It included a recommendation to perform a Phase 2 to confirm contamination, and the extent to which it may or may not require remediation. Phase 2 will run 10-20K depending on the number of wells drilled, and if you don't like the result and back out it's an expensive Due Diligence item.
Did your assessor make any recommendation, and if so, a quote how much that would cost? Could be good to ask about risk mitigation strategies like Innocent Owner Certification. Those also cost about 5K. This is why most people back away after a dirty Phase 1.