We consider distributed consensus in networks where the agents have integrator dynamics of order two or higher (n≥2). We assume all feedback to be localized in the sense that each agent has a bounded number of neighbors and consider a scaling of the network through the addition of agents in a modular manner, i.e., without re-tuning controller gains upon addition. We show that standard consensus algorithms, which rely on relative state feedback, are subject to what we term scale fragilities, meaning that stability is lost as the network scales. For high-order agents (n≥3), we prove that no consensus algorithm with fixed gains can achieve consensus in networks of any size. That is, while a given algorithm may allow a small network to converge, it causes instability if the network grows beyond a certain finite size. This holds in families of network graphs whose algebraic connectivity, that is, the smallest non-zero Laplacian eigenvalue, is decreasing towards zero in network size (e.g. all planar graphs). For second-order consensus (n=2) we prove that the same scale fragility applies to directed graphs that have a complex Laplacian eigenvalue approaching the origin (e.g. directed ring graphs). The proofs for both results rely on Routh–Hurwitz criteria for complex-valued polynomials and hold true for general directed network graphs. We survey classes of graphs subject to these scale fragilities, discuss their scaling constants, and finally prove that a sub-linear scaling of nodal neighborhoods can suffice to overcome the issue.