The apolipoprotein E (ApoE) ɛ4 allele is a well-established genetic risk factor for sporadic Alzheimer's disease. Some evidence suggests a negative role of the ApoE ɛ4 allele for cognitive performance in late life, while beneficial effects on cognition have been shown in young age. We investigated age-related effects of the ApoE gene on brain function by assessing cognitive performance, as well as functional activation patterns during retrieval of Face-Name pairs in a group of young (n = 50; age 26.4±4.6 years, 25 ɛ4 carriers) and old (n = 40; age 66.1±7.0 years, 20 ɛ4 carriers) participants. A cross-sectional factorial design was used to examine the effects of age, ApoE genotype, and their interaction on both cognitive performance and the blood oxygenation level dependent (BOLD) brain response during retrieval of Face-Name pairs. While there were no genotype-related differences in cognitive performance, we found a significant interaction of age and ApoE genotype on task-related activation bilaterally in anterior cingulate gyrus and superior frontal gyrus, as well as left and right insula. Old age was associated with increased activity in ɛ4 carriers. The increased BOLD response in old ɛ4 carriers during retrieval could indicate a neurocognitive disadvantage associated with the ɛ4 allele with increasing age. Furthermore, recruitment of neuronal resources resulted in enhanced memory performance in young ɛ4 carriers, pointing to a better neurofunctional capacity associated with the ApoE4 genotype in young age.