Diffuse axon injury is a common trauma that affects the axons in the brain's white matter. Computational models of axons, both in isolation and within the matrix, have been developed to study this injury at cellular and tissue levels. However, axonal behaviour depends strongly on the mechanical properties of the surrounding matrix. Accurate material properties of axons and the matrix are essential for realistic modelling of their behaviour. This study characterises the hyper-viscoelastic properties of axons and their matrix for human brain tissue in two different white matter regions. First, previous experimental data on isolated axons under tension were used to determine their mechanical properties. Then, employing finite element analysis, neural networks, and optimisation methods, matrix properties were inferred using experimental data on human brain tissue behaviour under three shear modes at large deformations and varying strain rates. The results indicate that axons are approximately 10–13 times stiffer than the surrounding matrix, depending on the region. The material properties defined in this study provide an accurate representation of axonal and matrix behaviour under injurious conditions, as they are based on large-strain and high-strain-rate data. The constitutive model can be used for a more precise assessment of the injury threshold and the mechanisms of diffuse axon injury at the cellular level.