In this paper, we investigate distributed multi-agent tracking of a convex set specified by multiple moving leaders with unmeasurable velocities. Various jointly connected interaction topologies of the follower agents with uncertainties are considered in the study of set tracking. Based on the connectivity of the time-varying multi-agent system, necessary and sufficient conditions are obtained for set input-to-state stability and set integral input-to-state stability for a nonlinear neighbor-based coordination rule with switching directed topologies. Conditions for asymptotic set tracking are also proposed with respect to the polytope spanned by the leaders.