We study the problem of learning without forgetting (LwF) in which a deep learning model learns new tasks without a significant drop in the classification performance on the previously learned tasks. We propose an LwF algorithm for multilayer feedforward neural networks in which we can adapt the number of layers of the network from the old task to the new task. To this end, we limit ourselves to convex loss functions in order to train the network in a layer-wise manner. Layer-wise convex optimization leads to low-computational complexity and provides a more interpretable understanding of the network. We compare the effectiveness of the proposed adaptive LwF algorithm with the standard LwF over image classification datasets.