Motivated by the limitations of conventional coarse-grained molecular dynamics for simulation of large systems of nanoparticles and the challenges in efficiently representing general pair potentials for rigid bodies, we present a method for approximating general rigid body pair potentials based on a specialized type of deep neural network that maintains essential properties, such as conservation of energy and invariance to the chosen origins of the particles. The network uses a specialized geometric abstraction layer to convert the relative coordinates of the rigid bodies to input more suitable to a conventional artificial neural network, which is trained together with the specialized layer. This results in geometric representations of the particles optimized for the specific potential. The network can be trained directly on scalar values to fit a model without explicit gradient and then used to efficiently evaluate the force and torque on the particles resulting from the potential. The concept is demonstrated with an atomistic interaction model for carbon nanotubes and the resulting model is compared with a common type of coarse-grained model optimized for the same potential, with even very small networks comparing favourably and larger networks achieving up to two orders of magnitude lower cost. The sensitivity to noise in the training data is investigated and the model is found to strongly reject noise up to 12.5% given a dataset of 107 samples. The performance of a proof-of-concept implementation is demonstrated on a variety of hardware, showing the models viability for large-scale simulations. Furthermore, generalization to soft bodies and potentials for polydisperse systems are discussed.