Service composition as an important technique for combining multiple services to construct a value-added service is a major research issue in cloud manufacturing. Highly dynamic environments present great challenges to cloud manufacturing service composition (CMfg-SC). Most of previous studies employ heuristic algorithms to solve service composition issues in cloud manufacturing, which, however, are designed for specific problems and lack adaptability necessary to dynamic environment. Hence, CMfg-SC calls for new adaptive approaches. Recent advances in deep reinforcement learning (DRL) provide a new means for solving this issue. Based on DRL, we propose a Deep Deterministic Policy Gradient (DDPG)-based service composition approach to cloud manufacturing, with which optimal service composition solutions can be learned through repeated training. Performance of DDPG in solving CMfg-SC in both static and dynamic environments is examined. Results obtained with another DRL algorithm -Deep Q-Networks (DQN) and the traditional Ant Colony Optimization (ACO) are also presented. Comparison indicates that DDPG has better adaptability, robustness, and extensibility to dynamic environments than ACO, although ACO converges faster and its steady QoS value of the service composition solution is higher than that of DDPG by 0.997%. DDPG outperforms DQN in convergence speed and stability, and the QoS value of the service composition solution of DDPG is higher than that of DQN by 3.249%.