An element of this is the expectation of privacy. I have an expectation of privacy in my own home, for example, but I have no such expectation when out in the town square. Unless your application can selectively disregard contacts in places where there's an expectation of privacy, it has a problem. And the device has no real way of knowing when that expectation exists.
There are lots of ways to work around this.
First, I probably should have been more clear, I'm not exclusively talking about this being a cell phone app. A simple camera network reading license plates on roads, highways, and public parking areas would be just as useful for many applications. And in the US most of these cameras already exist; they're just not being used for this purpose. Add in information from security check points in public buildings like airports and hospitals, and you've got even more data. Add in sci-fi level facial recognition and you've got more data than you could process.
On a phone app, though, there are still a lot of work-arounds with varying levels of grayness as to their morality and/or legality. The most obvious one being that software simply could selectively disregard information in places where there's an expectation of privacy. It's possible the phone may not know if something is public or private in certain inbetween areas, but it will also know when it's definitely in public areas such as the street or public buildings.
Alternately, just lower the resolution of the data. If your location voxel is larger than your house, it's implicit that it's not taking private data from inside your house. But you'll still be able to get plenty of useful tracking data even at low resolution.
Another option: don't continuously track a phone. Just place "checkpoints" at multiple places in public areas. Any time a phone is near the checkpoint, it connects and checks in. Since the checkpoints are only in public areas, there is no expectation of privacy. Still delivers plenty of data. This one also wouldn't require a phone. Conspiracy theorists have been talking about systems like this built into ID cards and credit cards and even dollar bills for decades. With RFID tags being reality instead of sci-fi, this is easier than ever.
The thesis of all this, which is moving farther and farther away from COVID19 as I type more examples, is that there are ways that the government could get large amounts of tracking data. The legal and technological barriers exist, but are only minor obstacles if someone had the proper motivation.