A gas bubble sitting at a liquid-gas interface can burst following the rupture of the thin liquid film separating it from the ambient, owing to the large surface energy of the resultant cavity. This bursting bubble forms capillary waves, a Worthington jet and subsequent droplets for a Newtonian liquid medium. However, rheological properties of the liquid medium like elastoviscoplasticity can greatly affect these dynamics. Using direct numerical simulations, this study exemplifies how the complex interplay between elasticity (in terms of elastic stress relaxation) and yield stress influences the transient interfacial phenomenon of bursting bubbles. We investigate how bursting dynamics depends on capillary, elastic and yield stresses by exploring the parameter space of the Deborah number ${{\textit {De}}}$ (dimensionless relaxation time of elastic stresses) and the plastocapillary number $\mathcal {J}$ (dimensionless yield-stress of the medium), delineating four distinct characteristic behaviours. Overall, we observe a non-monotonic effect of elastic stress relaxation on the jet development while plasticity of the elastoviscoplastic (EVP) medium is shown to affect primarily the jet evolution only at faster relaxation times (low ${{\textit {De}}}$). The role of elastic stresses on jet development is elucidated with the support of energy budgets identifying different modes of energy transfer within the EVP medium. The effects of elasticity on the initial progression of capillary waves and droplet formation are also studied. In passing, we study the effects of solvent-polymer viscosity ratio on bursting dynamics and show that polymer viscosity can increase the jet thickness apart from reducing the maximum height of the jet.