Both healthy and pathological aging due to Alzheimer's disease (AD) are associated with decreased brain grey matter volume (GMV) and disrupted white matter (WM) microstructure. Thinner macular ganglion cell-inner plexiform layer (GC-IPL) measured by spectral-domain optical coherence tomography has been reported in patients with AD and mild cognitive impairment. Emerging evidence suggested a link between thinner GC-IPL and lower GMV in subjects with no dementia using region-of-interest-based approach. However, it remains unknown whether GC-IPL thickness is associated with brain WM microstructure and how such association differed between normal and cognitively impaired subjects. Here, for subjects with no cognitive impairment (NCI), thinner GC-IPL was associated with lower WM microstructure integrity in the superior longitudinal fasciculus, inferior fronto-occipital fasciculus, corticospinal tracts, anterior thalamic radiation, and cingulum regions, while it was weakly associated with lower GMV in visual cortex and cerebellum. Nevertheless, these retina-brain associations were disrupted in the presence of cognitive impairment. Correlations between GMV and GC-IPL were lost in patients with cognitive impairment but no dementia (CIND) and AD patients. GC-IPL was related to WM microstructural disruption in similar regions with decreased significance. In contrast, lower WM microstructure integrity in the fornix showed a trend of correlation with thinner GC-IPL in both CIND and AD but not NCI. Collectively, our findings suggest the possible physiological retina-brain relationship in healthy aging, which might be disrupted by disease-induced changes in patients with cognitive impairment. Longitudinal study with larger patient sample should follow to confirm the disease mechanism behind these retina-brain relationship changes.