When observing a dynamic world, the size of image structures may vary over time. This article emphasizes the need for including explicit mechanisms for automatic scale selection in feature tracking algorithms in order to: (i) adapt the local scale of processing to the local image structure, and (ii) adapt to the size variations that may occur over time. The problems of corner detection and blob detection are treated in detail, and a combined framework for feature tracking is presented. The integrated tracking algorithm overcomes some of the inherent limitations of exposing fixed-scale tracking methods to image sequences in which the size variations are large. It is also shown how the stability over time of scale descriptors can be used as a part of a multi-cue similarity measure for matching. Experiments on real-world sequences are presented showing the performance of the algorithm when applied to (individual) tracking of corners and blobs.