Conventional fatigue assessment of steel railway bridges relies on strain monitoring and cycle counting (e.g., rainflow) to obtain stress spectra. Motivated by accelerometers’ long-term stability and ease of installation, recent studies seek to infer local stress histories from bridge vibrations; however, this remains challenging because acceleration signals are noisy and confounded by structural dynamics, variable loading, resonance, and train–track interactions. To address these challenges, this study introduces Vib2StressNet, a deep-learning architecture that maps multi-channel vertical vibration signals directly to fatigue stress spectra. Unlike sequence-to-sequence models that reconstruct full time histories, this approach bypasses intermediate steps to focus on damage accumulation. The architecture integrates convolutional layers with a multi-head self-attention mechanism that captures information across multiple frequency bands. A critical design feature is the inclusion of train speed as an auxiliary scalar embedding, allowing the network to dynamically adapt to speed-dependent resonance. Vib2StressNet demonstrated strong generalizability across three Swedish railway bridges with diverse structural configurations, train types, and dynamic loading conditions. Under resonance and variable-speed conditions, adding train speed significantly improved prediction accuracy, reducing the mean squared error (MSE) by more than 20%. Relative to a previous sequence-to-sequence baseline model that reconstructs stresses before rainflow counting, Vib2StressNet simplifies the workflow and reduces MSE from 59.8 to 0.34 with more test samples. Interpretability analyses further confirm that preserving high-frequency components associated with axle impacts improves accuracy. These findings establish Vib2StressNet as a cost-effective tool for long-term monitoring that supports predictive maintenance without permanent strain instrumentation.