The significant growth in data consumption among mobile users necessitates the development of new architecture to meet the increasing demand. On the other hand, reconfigurable intelligent surface (RIS) has grown in popularity in 6G due to its improved spectral efficiency, simplicity of deployment, and low cost. However, with the constrained limitation of the coverage by conventional RIS, the research direction has turned towards simultaneously transmitting and reflecting RIS (STAR-RIS) to provide 360° coverage alongside the benefits of RIS. In this paper, a STAR-RIS-assisted downlink communication system for both indoor and outdoor users is investigated. Then, the optimization problem to maximize the spectral efficiency while jointly controlling the beamforming power for each user and phase shift values of the STAR-RIS is formulated. Since the formulated problem is NP-hard and challenging to solve in polynomial time, a policy gradient method for reinforcement learning named proximal policy optimization (PPO) is implemented to solve the problem. To demonstrate the effectiveness of our proposed algorithm, extensive simulation results are executed. Numerical results prove that our proposed algorithm outperforms several benchmark schemes in the literature.