With the accelerated integration of connected and autonomous vehicles (CAVs) into transportation systems, such vehicles have evolved rapidly. Emerging studies have focused on the considerations influencing travelers’ acceptance of CAVs and the factors contributing to travel choice intention. However, the different reasons for the diffusion of CAVs and the associated trends in relation to this are key questions that still require investigation. Therefore, this study establishes a structural equation modeling (SEM) framework to explore the impact of both external factors and internal subjective factors on travel choice. Based on the influencing factors identified by SEM, this study develops an agent-based model simulation approach to examine the diffusion trends in relation to CAV travelers from an individual perspective. To calibrate the model, a questionnaire survey is designed to obtain data from travelers in Shenzhen, Guangdong Province, China. The survey results show that CAV choice intention is influenced by individual (including innovativeness), travel-related, and social influence factors. The simulation experiments reveal that the diffusion of CAV travelers is a complex process. The lower cost of CAV travel has a positive impact initially and midway through the process of diffusion in relation to CAV users. For example, in situations in which CAVs are more cost-effective, during the simulation time (years) the number of CAV users fluctuates significantly between the 7th and 30th years. There is a notable increase of 10% in CAV users in the 15th year and, eventually, the total number of CAV users stabilizes at 71.8% of all travelers. The findings will assist agencies and CAV operators to implement effective promotion strategies.