An increasing number of cities around the globe aim to become Smart Cities through the implementation of a range of Information and Communication technologies (ICT), AI applications and cloud-based IoT solutions. While the underlying motivations vary, all such transitions require large amounts of data. In this paper, we articulate and defend two claims about privacy in a Smart City. Our first claim is that some level of systematic data collection and processing is ethically permissible. However, there is an upper limit for what is permissible: We suggest that it is never permissible to collect and process data that significantly undermine people's autonomy. Our second claim specifies when such minor privacy infringements that do not significantly undermine people's autonomy are permissible: We suggest that the only policies legitimized by the first claim are those that promote the collective good.