During the last decade, model based tracking of objects and its necessity in visual servoing and manipulation has been advocated in a number of systems [4], [7], [9], [12], [13], [14]. Most of these systems demonstrate robust performance for cases where either the background or the object are relatively uniform in color. In terms of manipulation, our basic interest is handling of everyday objects in domestic environments such as a home or an office. In this paper, we consider a number of different parameters that effect the performance of a model-based tracking system. Parameters such as color channels, feature detection, validation gates, outliers rejection and feature selection are considered here and their affect to the overall system performance is discussed. Experimental evaluation shows how some of these parameters can successfully be evaluated (learned) on-line and consequently improve the performance of the system.