Personalized music playlists are increasingly being used in health-care contexts to address the psychological and behavioral symptoms in people with dementia. However, there is little understanding of how people with different mental health histories and symptoms respond differently to music. A factorial experiment was conducted to investigate the influence of depression, anxiety, apathy, and cognitive decline on affective response to music. Ninety-nine people with dementia listened to three music playlists based on personal preferences. Activation of facial action units was measured, and behavioural responses continuously observed. Results demonstrated that people with high levels of depression and with symptoms of Alzheimer's type dementia demonstrated increased levels of sadness when listening to music. People with low depression but high levels of apathy demonstrated the highest behavioral evidence of pleasure during music listening, although behavioral evidence declined with severity of cognitive impairment. It is concluded that as well as accounting for personal preferences, music interventions for people with dementia need to take mental health history and symptoms into account.