Generalisation across image scales remains a fundamental challenge for deep networks, which often fail to handle images at scales not seen during training (the out-of-distribution problem). In this paper, we present provably scale-invariant Gaussian derivative residual networks (GaussDerResNets), constructed out of scale-covariant Gaussian derivative residual blocks coupled in cascade, aimed at addressing this problem.

By adding residual skip connections to the previous notion of Gaussian derivative layers, deeper networks with substantially increased accuracy can be constructed, while preserving very good scale generalisation properties at the higher level of accuracy. Explicit proofs are provided regarding the underlying scale-covariant and scale-invariant properties in arbitrary dimensions. We also conceptually relate the functionality of the Gaussian derivative residual blocks to semi-discretisations of the velocity-adapted affine diffusion equation.

To analyse the ability of GaussDerResNets to generalise to new scales, we apply them on the new rescaled version of the STL-10 dataset, where training is done at a single fixed scale and evaluation is performed on multiple copies of the test set, each rescaled to a single distinct spatial scale, with scale factors extending over a range of 4. We also conduct similar systematic experiments on the rescaled versions of Fashion-MNIST and CIFAR-10 datasets introduced in our previous work.

Experimentally, we demonstrate that the GaussDerResNets have strong scale generalisation and scale selection properties, while also achieving good test accuracy, on all the three rescaled datasets. In our ablation studies, we investigate different architectural variants of GaussDerResNets, demonstrating that basing the architecture on depthwise-separable convolutions allows for decreasing both the number of parameters and the amount of computations, with reasonably maintained accuracy and scale generalisation. We also find that including a zero-order Gaussian term in the layer definition can sometimes be beneficial, as demonstrated for ourspatial-max-pooling-based networks trained on the rescaled STL-10 dataset.

In these ways, we demonstrate how deep networks can in a theoretically well-founded way handle variations in scale in the testing data that are not spanned by the training data.