Prediction-driven approach to model selection using feature selection and nonrandom hold-out validation
(English)Manuscript (preprint) (Other academic)
In this paper, we address the forecast problem and propose a prediction drivenapproach to model selection. Furthermore, we investigate to what extent dierentprediction questions lead to dierent \best" models. Most of the studies in theeld, take an inference-driven approach to select the best model and project theresults to the future population. In contrast, we take a prediction-driven approachfor both selection criteria and validation sample. We use feature (variable) selectionand nonrandom hold-out validation to select the model with the highest predictiveperformance in an out-of-sample prediction manner. The application of interest iscar type choice modeling using the Swedish car eet data. We introduce two dier-ent types of model selection criteria: maximum likelihood which is the conventionalmethod of model selection, and root mean squared error of the prediction quantityof interest. We compare the best models obtained by dierent criterion functions.The results show that the \best" model for the purpose of prediction depends con-siderably on the prediction question of interest. Moreover, when the objective isto predict a sub-section of a population such as the total share of ethanol cars,maximizing log-likelihood is not the most accurate model selection criterion.
model selection, nonrandom hold-out validation, feature selection, out of sample prediction, car type choice models
Research subject Transport Science
IdentifiersURN: urn:nbn:se:kth:diva-180341OAI: oai:DiVA.org:kth-180341DiVA: diva2:892919
QS 20162016-01-112016-01-112016-01-15Bibliographically approved