Millimeter wave (mmWave) communication can realize high transmission rates in vehicular networks. Nevertheless, severe blocking effects and high mobility of vehicles would seriously affect downlink services for vehicles. To ensure communication quality and stability, this paper jointly explores beam allocation and relay selection in mmWave vehicular networks from the perspective of artificial intelligence-driven model. We utilize queuing theory to simulate dynamic distributions of vehicles and firstly propose a deep reinforcement learning (DRL) based joint beam allocation and relay selection scheme to mitigate the blocking effects and optimize the total communication capacity. When the expected downlink is blocked, mmWave base station (mmBS) can select appropriate idle vehicles as the relay nodes for service. Besides, we set the capacity threshold when designing the scheme to guarantee each target vehicle can obtain the ideal service. Through proper training, mmBS can intelligently find an optimal solution for the constantly updated vehicular networks based on the location of vehicles. Simulation results demonstrate the effectiveness of our scheme, which can restrain the transmission outage caused by random blockage and improve the total communication capacity of vehicular networks.