In the US there is legislation called, "‘Freedom to Display the American Flag Act of 2005." You can thank Bush for that one. Furthermore not permitting the display a Purple Heart military flag may be construed as an FHA violation for discrimination against someone with a disability. Same for a flag of Jesus or Prophet Mohammed or the Scientology Robot King. You get the gist. It's not quite as simple as banning everything.
I am leaning more and more towards a detailed rule that allows the flying of flags of a certain size, in certain locations, being maintained a certain way. Consistent enforcement is the key.
If someone tries to get clever and put up a Nazi or ISIS flag, while adhering to all the aforementioned then maybe an FBI and law enforcement inquiry would be worth reporting. I suspect the neighbors would take action in many of these cases before I would have to get involved.
There is typically more good to be had from flag flying than bad. A few bad apples shouldn't ruin it for everyone.