Hybrid electric vehicles (HEVs) rely on energy management strategies (EMSs) to achieve optimal fuel economy. However, both model- and learning-based EMSs have their respective limitations which negatively affect their performances in online applications. This paper presents a computationally efficient adaptive dynamic programming (ADP) approach that can not only rapidly calculate optimal control actions but also iteratively update the approximated value function (AVF) according to the actual fuel and electricity consumption with limited computation resources. Exploiting the AVF, the engine on/off switch and torque split problems are solved by one-step lookahead approximation and Pontryagin's minimum principle (PMP), respectively. To raise the training speed and reduce the memory space, the tabular value function (VF) is approximated by carefully selected piecewise polynomials via the parametric approximation. The advantages of the proposed EMS are threefold and verified by processor-in-the-loop (PIL) Monte Carlo simulations. First, the fuel efficiency of the proposed EMS is higher than that of an adaptive PMP and close to the theoretical optimum. Second, the new method can adapt to the changed driving conditions after a small number of learning iterations and thus has higher fuel efficiency than a non-adaptive dynamic programming (DP) controller. Third, the computation efficiencies of the proposed AVF and a tabular VF are compared. The concise data structure of the AVF enables faster convergence and saves at least 70% of onboard memory space without obviously increasing the average CPU utilization.