We study the problem of distributed consensus in networks where the local agents have high-order (n ≥ 3) integrator dynamics, and where all feedback is localized in that each agent has a bounded number of neighbors. We prove that no consensus algorithm based on relative differences between states of neighboring agents can then achieve consensus in networks of any size. That is, while a given algorithm may allow a small network to converge to consensus, the same algorithm will lead to instability if agents are added to the network so that it grows beyond a certain finite size. This holds in classes of network graphs whose algebraic connectivity, that is, the smallest non-zero Laplacian eigenvalue, is decreasing towards zero in network size. This applies, for example, to all planar graphs. Our proof, which relies on Routh-Hurwitz criteria for complex-valued polynomials, holds true for directed graphs with normal graph Laplacians. We survey classes of graphs where this issue arises, and also discuss leader-follower consensus, where instability will arise in any growing, undirected network as long as the feedback is localized.