Earth observation satellites play a crucial role in disaster response and management, offering timely and large-scale data for damage assessment. Recent studies have demonstrated the potential of deep learning techniques for automated building damage detection from satellite imagery, often based on the xBD dataset. This high-quality dataset features bi-temporal very-high-resolution image pairs of several disaster events. Notably, several studies have proposed new network architectures and demonstrated their improved performance on xBD. Although such highly engineered model-centric approaches achieve promising results on the original dataset split of xBD, we show that they underperform on a new event-based split, which evaluates them on unseen events. To reduce this generalization gap, we propose to follow a data-centric approach. For this, we first derive a simplified baseline method from the winning solution of the xView2 competition, with greatly reduced complexity. With a simple adjustment to this baseline method, we incorporate readily available disaster-type information, allowing it to account for disaster-specific damage characteristics. We evaluate the resulting disaster-adaptive model on the event-based split of xBD and demonstrate its improved ability to generalize to unseen events compared to several competing methods. These results highlight the potential of our data-centric approach for practical and robust building damage assessment in real-world disaster scenarios. Code including the strong baseline model is available at: https://github.com/SebastianHafner/DisasterAdaptiveNet.