Structural balance theory has been developed in sociology and psychology to explain how interacting agents, e.g., countries, political parties, opinionated individuals, with mixed trust and mistrust relationships evolve into polarized camps. Recent results have shown that structural balance is necessary for polarization in networks with fixed, strongly connected neighbor relationships when the opinion dynamics are described by DeGroot-type averaging rules. We develop this line of research in this paper in two steps. First, we consider fixed, not necessarily strongly connected, neighbor relationships. It is shown that if the network includes a strongly connected subnetwork containing mistrust, which influences the rest of the network, then no opinion clustering is possible when that subnetwork is not structurally balanced; all the opinions become neutralized in the end. In contrast, it is shown that when that subnetwork is indeed structurally balanced, the agents of the subnetwork evolve into two polarized camps and the opinions of all other agents in the network spread between these two polarized opinions. Second, we consider time-varying neighbor relationships. We show that the opinion separation criteria carry over if the conditions for fixed graphs are extended to joint graphs. The results are developed for both discrete-time and continuous-time models.