Variations in the behavior inherent to a structure can occur when the environmental (e.g. temperature) and operational (e.g. loading) conditions change. But variations in the behavior can also occur due to adverse events such as existing damage. Thus, a monitoring system that takes dynamic properties as damage-sensitive features needs foremost to comprehend the natural variability of these properties, which are often larger and can be misunderstood as changes prompted by damage. This leads to wrong conclusions during the assessment of the structure's health condition that can end in wrong decision-making with substantial unnecessary costs. In this paper an ANN-based method for damage detection is proposed and validated with monitored data from an existing healthy bridge - a single span, ballasted railway bridge. Temperatures at the bridge location oscillate year-round between -30°C and +30°C. Some of the recorded healthy data sets will be used to train the ANN to learn the reference state of the structure. In the absence of recorded damaged data, the remaining recorded healthy data sets can then be used to test the ANN. Due to the large spread in temperatures, the testing data sets can describe a very different behavior of the bridge when compared to the training data sets, even though the healthy condition has remained over the duration of the monitoring period. That difference in behavior can be wrongly interpreted as damage. The results prove, based on a case study of a real bridge, that seasonal effects cannot be dismissed when designing and developing a damage detection system to perform efficiently.