Not only is your property grandfathered, but your setbacks are as well. The city cannot pick and choose portions of their zoning ordinances to apply to your property as the usage is non-conforming.
It’s time for you to get a municipal lawyer to have an initial conversation with their attorney about how grandfathering works. In virtually every state in the USA grandfathering laws have held up when cities get overzealous like this. In a lot of cases this will get worked out when you get those guys talking to each other.
Here’s a good post from Frank on the topic: Putting the grandfathering concept to bed once and for all