Cloud manufacturing is a service-oriented manufacturing model that offers manufacturing resources as cloud services. Robots are an important type of manufacturing resources. In cloud manufacturng, large-scale distrib-uted robots are encapsulated into cloud services and provided to consumers in an on-demand manner. How to effectively and efficiently manage and schedule decentralized robot services in cloud manufacturing to achieve on-demand provisioning is a challenging issue. During the past few years, Deep Reinforcement Learning (DRL) has become very popular and successfully been applied to many different areas such as games, robotics, and manufacturing. DRL also holds tremendous potential for solving scheduling issues in cloud manufacturing. To this end, this paper is devoted to exploring effective approaches for scheduling of decentralized robot manufacturing services in cloud manufacturing with DRL. Specifically, both Deep Q-Networks (DQN) and Dueling Deep Q-Networks (DDQN)-based scheduling algorithms are proposed. Performance of different algo-rithms, including DQN, DDQN, and other three benchmark algorithms, indicates that DDQN performs the best with respect to each indicator. Effects of different combinations of weight coefficients and influencing degrees of different indicators on the overall scheduling objective are analyzed. Results indicate that the DDQN-based scheduling algorithm is able to generate scheduling solutions efficiently.