In this paper, we present a framework for a robotic system with the ability to perform real-world manipulation tasks. The complexity of such tasks determines the precision and freedoms controlled which also affects the robustness and the flexibility of the system. The aspect is on the development of visual system and visual tracking techniques in particular. Since precise tracking and control of a full pose of the object to be manipulated is usually less robust and computationally expensive, we integrate vision and control system where the objectives are to provide the discrete state information required to switch between control modes of different complexity. For this purpose, an integration of simple visual algorithms is used to provide a robust input to the control loop. Consensus theory is investigated as the integration strategy. In addition, a general purpose framework for integration of processes is used to implement the system on a real robot. The proposed approach results in a system which can robustly locate and grasp a door handle and then open the door.