The distributed microgrids cooperate to accomplish economic and environmental objectives, which have a vital impact on maintaining the reliable and economic operation of power systems. Therefore a distributed multi-agent reinforcement learning (MARL) algorithm is put forward incorporating the actor-critic architecture, which learns multiple critics for subtasks and utilizes only information from neighbors to find dispatch strategy. Based on our proposed algorithm, multi-objective optimal dispatch problem of microgrids with continuous state changes and power values is dealt with. Meanwhile, the computation and communication resources requirements are greatly reduced and the privacy of each agent is protected in the process of information interaction. In addition, the convergence for the proposed algorithm is guaranteed with the adoption of linear function approximation. Simulation results validate the performance of the algorithm, demonstrating its effectiveness in achieving multi-objective optimal dispatch in microgrids.