This paper considers the safety-critical control problem for nonlinear second-order multi-agent systems with constraints of each agent and inter-agent ones. We overcome the challenge of the time-varying and position-dependent communication network with limited sensing range by introducing a truncated function for the smooth addition and deletion of links in the edge set, and design a distributed and locally Lipschitz-continuous safety-critical control law, composed of a nominal controller for the objectives such as consensus, formation, and position swapping, etc., and a safety controller, which only takes effect when some neighboring agent enters the custom-designed boundary set. Meanwhile, to rigorously verify the safety of the whole multi-agent system, a continuously differentiable control barrier function is proposed under a relaxed feasibility condition in the sense that it is imposed on each subsystem and only needed in the boundary area.