The association between cerebrovascular disease pathology (measured by white matter hyperintensities, WMH) and brain atrophy in early Alzheimer's disease (AD) remain to be elucidated. Thus, we investigated how WMH influence neurodegeneration and cognition in prodromal and clinical AD. We examined 51 healthy controls, 35 subjects with mild cognitive impairment (MCI), and 30 AD patients. We tested how total and regional WMH is related to specific grey matter volume (GMV) reductions in MCI and AD compared to controls. Stepwise regression analysis was further performed to investigate the association of GMV and regional WMH volume with global cognition. We found that total WMH volume was highest in AD but showed the strongest association with lower GMV in MCI. Frontal and parietal WMH had the most extensive influence on GMV loss in MCI. Additionally, parietal lobe WMH volume (but not hippocampal atrophy) was significantly associated with global cognition in MCI while smaller hippocampal volume (but not WMH volume) was associated with lower global cognition in AD. Thus, although WMH volume was highest in AD subjects, it had a more pervasive influence on brain structure and cognitive impairment in MCI. Our study thus highlights the importance of early detection of cerebrovascular disease, as its intervention at the MCI stage might potentially slow down neurodegeneration.