The white matter tract-specific correlates of neuropsychological deficits are not fully established in patients with subcortical vascular cognitive impairment (SVCI), where white matter tract damage may be a critical factor in cognitive impairment. The purpose of this study is to investigate the tract-specific correlates of neuropsychological deficits in SVCI patients using tract-specific statistical analysis (TSSA). We prospectively recruited 114 SVCI patients, and 55 age-, gender-, and education-matched individuals with normal cognition (NC). All participants underwent diffusion weighted imaging and neuropsychological testing. We classified tractography results into fourteen major fiber tracts and analyzed group comparison and correlation with cognitive impairments. Relative to NC subjects, SVCI patients showed decreased fractional anisotropy values in bilateral anterior-thalamic radiation, cingulum, superior-longitudinal fasciculus, uncinate fasciculus, corticospinal tract, and left inferior-longitudinal fasciculus. Focal disruptions in specific tracts were associated with specific cognitive impairments. Our findings suggest that disconnection of specific white matter tracts, especially those neighboring and providing connections between gray matter regions important to certain cognitive functions, may contribute to specific cognitive impairments in SVCI.