Perceived neighborhood safety significantly affects the quality of life and social cohesion of communities. However, the perspectives of non-residents are often neglected, despite their potential influence on local views. This study addresses this gap by examining how locals (residents and frequent visitors) and outsiders (non-visitors) perceive safety in the largely stigmatized Stockholm district of Järva. We use a digital participatory mapping tool that allows citizens to share safety perceptions and experiences of victimization interactively. Surveying 592 respondents and using a mixed-methods approach, the study investigates the environmental, social, and emotional factors shaping these perceptions. Findings highlight how stigma, place attachment, and familiarity play an important role in moderating safety concerns, as even minor first-hand experience with Järva was linked to improved neighborhood perception. The study concludes by reflecting upon the importance of addressing territorial stigma and promoting local engagement to improve local and outsider safety perceptions.