This paper investigates the stability of discrete-time linear systems with stochastic delays. We assume that delays are modeled as random variables, which take values in integers with a certain probability. For the scalar case, we provide an analytical bound on the probability to guarantee the stability of linear systems. In the vector case, we derive a linear matrix inequality condition to compute the probability for ensuring the stability of closed-loop systems. As a special case, we also determine the step size of gradient algorithms with stochastic delays in the unconstrained quadratic programming to guarantee convergence to the optimal solution. Numerical examples are provided to show the effectiveness of the proposed analysis techniques.