Multi-Gbps data rates are achievable in millimeter-wave (mmWave) bands, but a prominent issue is the tiny wavelength that results in rapid fading variations and significant pilot signaling for channel estimation. In this letter, we recognize that the angles of scattering clusters seen from the UE vary slowly compared to the small-scale fading. We characterize the beam coherence time, which quantifies how frequently the UE must update its downlink receive combining matrix. The exact beam coherence time is derived in the single-cluster case, and an achievable lower bound is proposed for the multi-cluster case. These values are determined so that at least half of the received signal gain is maintained in between the combining updates. We demonstrate how the beam coherence time can be hundreds of times larger than the channel coherence time of the small-scale fading.