The existing usage dictates what you can do with the lots. If you have 6 inch setbacks between homes on one side today then that would prevail under grandfathering when you move in additional homes on vacant lots.
The big question is does the city agree with this or does your attorney have to help them understand it? Work this out before closing, and if you have to spend more than 500 bucks then the Seller needs to cover it or you back out.