Rare critical scenarios are crucial to verify the performance of autonomous driving in different situations. Critical scenario generation requires the ability of sampling critical combinations from an infinite parameter space in the logical scenario. Existing solutions aim to explore the correlation of action parameters in the initial scenario rather than action sequences. How to model action sequences so that one can further consider the effects of different action parameters is the bottleneck of the problem. In this paper, we solve the problem by proposing the ECSAS framework. Specifically, we first propose a description language, BTScenario, allowing us to model action sequences of scenarios. We then use reinforcement learning to search for combinations of critical action parameters. Several optimizations are proposed to increase efficiency, including action mask and replay buffer. Experimental results show that our model with strong collision ability and effectively outperforms the existing methods on various nontrivial scenarios.