In this paper, we present a system for continuous posture recognition. The main contributions of the proposed approach are the integration of an adaptive color model with a tracking system that allows for robust continuous posture recognition based on Principal Component Analysis. The adaptive color model uses Gaussian Mixture Models for skin and background color representation, Bayesian framework for classification and Kalman filter for tracking hands and head of a person that interacts with the robot. Experimental evaluation shows that the integration of tracking and an adaptive color model supports the robustness and flexibility of the system when illumination changes occur.