We present a sample-based approach for tuning an anomaly detector threshold to achieve an acceptable false alarm rate without a priori knowledge of system or detector dynamics. If the distribution of the output of the detector is known, finding such a threshold can be re-interpreted as determining a specific quantile of the detector output distribution, which is the minimizer of a convex optimization problem. The sample-based approach we propose approximates the threshold from the empirical distribution. We, further, identify distribution free finite sample guarantees that give the number of samples required to ensure the false alarm rate is near the acceptable value. Finally, we numerically verify our approach on both static and dynamic anomaly detectors, where we investigate both light- and heavy-tailed distributions.