The aim of this study is to investigate and compare how 12-weeks of Tai Chi Chuan and Baduanjin exercise can modulate brain structure and memory function in older adults. Magnetic resonance imaging and memory function measurements (Wechsler Memory Scale-Chinese revised, WMS-CR) were applied at both the beginning and end of the study. Results showed that both Tai Chi Chuan and Baduanjin could significantly increase grey matter volume (GMV) in the insula, medial temporal lobe, and putamen after 12-weeks of exercise. No significant differences were observed in GMV between the Tai Chi Chuan and Baduanjin groups. We also found that compared to healthy controls, Tai Chi Chuan and Baduanjin significantly improved visual reproduction subscores on the WMS-CR. Baduanjin also improved mental control, recognition, touch, and comprehension memory subscores of the WMS-CR compared to the control group. Memory quotient and visual reproduction subscores were both associated with GMV increases in the putamen and hippocampus. Our results demonstrate the potential of Tai Chi Chuan and Baduanjin exercise for the prevention of memory deficits in older adults.