Transom wave behind planing hulls is a complicated physical phenomenon that has lead to computational challenge for many researchers. On the other hand, smoothed particle hydrodynamics (SPH) which is known as a meshless Lagrangian approach can simulate free surface flows with strongly nonlinear physics. Therefore, effort has been made in the current study to develop a 3D-SPH code for three-dimensional simulation of transom stern flow behind a rectangular planing hull. It is also aimed to give some new physical insights into this highly nonlinear problem. Different techniques such as sub particle scale turbulence model and moving least square density filter among others are also implemented. To validate the developed 3D-SPH code, the benchmark problem of dam breaking is investigated. Moreover, to verify the capability of the presented SPH model for transom flow simulation, previous experimental studies at low Froude numbers are considered. Comparisons display good agreement between the numerical results and experimental findings. Furthermore, a detailed discussion about rooster tail formation is presented.