As typical prosumers, commercial buildings equipped with electric vehicle (EV) charging piles and solar photovoltaic panels require an effective energy management method. However, the conventional optimization-model-based building energy management system faces significant challenges regarding prediction and calculation in online execution. To address this issue, a long short-term memory (LSTM) recurrent neural network (RNN) based machine learning algorithm is proposed in this paper to schedule the charging and discharging of numerous EVs in commercial-building prosumers. Under the proposed system control structure, the LSTM algorithm can be separated into offline and online stages. At the offline stage, the LSTM is used to map states (inputs) to decisions (outputs) based on the network training. At the online stage, once the current state is input, the LSTM can quickly generate a solution without any additional prediction. A preliminary data processing rule and an additional output filtering procedure are designed to improve the decision performance of LSTM network. The simulation results demonstrate that the LSTM algorithm can generate near-optimal solutions in milliseconds and significantly reduce the prediction and calculation pressures compared with the conventional optimization algorithm.