Cognitive impairment and dementia are common contributors to institutionalization and loss of quality of life in older people. Both type 2 diabetes mellitus (T2DM) and physical inactivity are prevalent and important modifiable risk factors for developing dementia. Physical activity is recommended in the management of T2DM, and there is growing evidence that exercise, a subgroup of physical activity, is also beneficial for maintaining and improving brain structure and function. This paper reviews the evidence for a benefit of exercise on T2DM related cognitive impairment and dementia. In addition, the type (e.g., aerobic, resistance), intensity, duration, and frequency of exercise are discussed. This review shows that although exercise has known benefits on the mechanisms linking T2DM to dementia, there are very few randomized controlled trials examining whether this is the case. It is concluded that the uptake of exercise for the brain has great potential to improve quality of life and provide significant cost savings, but further research is warranted to clarify the effects of exercise on T2DM and those on dementia.