Background: The adoption of eHealth accelerated during the COVID-19 pandemic. Inequalities in the adoption of eHealth during the COVID-19 pandemic have been reported, but there are few such studies among people with impairment. Objectives: This study aimed to investigate self-reported use and difficulty in the use of eHealth before the COVID-19 pandemic compared to during late social distancing restrictions in Sweden, among people with and without impairment, as well as between different types of impairment. Methods: A cross-sectional survey was distributed twice by snowball sampling to people with self-reported impairment and a general population matched by age, gender, and county. Use and difficulty in the use of six eHealth services were compared between groups using chi-square test and logistic regression with year interaction terms, reported as odds ratio adjusted (aOR) for gender and age with 95% CI. Results: The surveys included 1631 (in 2019) and 1410 (in 2021) participants with impairment, and 1084 (in 2019) and 1223 (in 2021) participants without. Participants with impairment, compared to those without impairment, reported less use and more difficulty in booking health care appointments online, digital identification, and the Swedish national web portal for health information and eHealth services (1177.se), both before and during the pandemic (P = .003 or lower). Video health care appointments were the exception to this disability digital divide in eHealth as video appointment adoption was the most likely among participants with attention, executive, and memory impairments (interaction term aOR 2.10, 95% CI 1.30‐3.39). Nonuse and difficulty in the use of eHealth were consistently associated with language impairments and intellectual impairments. For example, language impairments were inversely associated with use of the logged-in eHealth services in 1177.se in 2021 (aOR 0.49, 95% CI 0.36‐0.67) and were associated with difficulty in the use of 1177.se in 2019 (aOR 2.24, 95% CI 1.50‐3.36) and the logged-in eHealth services in 1177.se in 2021 (aOR 1.89, 95% CI 1.32‐2.70). Intellectual impairments were inversely associated with the use of the logged-in eHealth services in 1177.se in 2021 (aOR 0.19, 95% CI 0.13‐0.27). Conclusions: This repeated cross-sectional survey study, including participants with diverse types of impairment and a control group without impairment, reveals persisting disability digital divides, despite an accelerated adoption of eHealth across the pandemic. eHealth services were not accessible to some groups of people who were identified as being at risk of severe disease during the COVID-19 pandemic. This implies that all people could not use eHealth as a measure of infection protection.