This paper studies the consensus problem of multi-agent systems in which all agents are modeled by a general linear system. The authors consider the case where only the relative output feedback between the neighboring agents can be measured. To solve the consensus problem, the authors first construct a static relative output feedback control under some mild constraints on the system model. Then the authors use an observer based approach to design a dynamic relative output feedback control. If the adjacent graph of the system is undirected and connected or directed with a spanning tree, with the proposed control laws, the consensus can be achieved. The authors note that with the observer based approach, some information exchange between the agents is needed unless the associated adjacent graph is completely connected.