Objective. Proton therapy treatment planning currently needs to account for relatively large range uncertainty margins primarily due to the semi-empirical calibration of the treatment planning x-ray computed tomography (CT) to proton stopping power relative to water (RSP). Proton radiography enables direct measurement of integral RSP, offering potential to improve calibration and reduce these uncertainties. Approach. Three deep neural network architectures are trained to infer a patient RSP map using the treatment planning CT and proton radiographies, assuming straight proton trajectories. The best suited architecture is then evaluated on more realistic, clinical-like data generated with Monte Carlo simulations. Three idealized proton imaging detectors are simulated: single particle tracking (SPT), a pixelated energy-resolving imager (PERI) and a proton-integrating detector (PID). Main results. The learned primal dual (LPD) architetcure performs best in the simplified imaging scenario. In the more realistic scenario, the initial calibration error (median absolute percentage error of 2.24% across the test set) is reduced using only two projections across all detector types. SPT and PERI reach similar performance (1.10%/1.12%), followed by PID (1.30%). Restricting the LPD to the calibration task by incorporating prior knowledge of the functional relationship of Hounsfield Units (HUs) and RSP further improves calibration performance. For SPT, conventional optimization on detector data acquired for individual protons outperformed the data-driven method (0.16% vs. 1.10%). However, for PERI and PID, the data-driven approach (1.12%/1.30%) slightly outperformed conventional optimization (1.63%/1.72%). Significance. To our knowledge, this is the first study to apply a deep learning-based approach fusing proton radiographies and treatment planning CT data for improved RSP calibration. The method achieves lower calibration errors than idealized, conventional calibration curve optimization on PERIs and PIDs-detector types that offer a promising path for clinical adoption due to their lower complexity and cost compared to SPT systems.