Diagnostic distinction of primary progressive aphasias (PPA) remains challenging, in particular for the logopenic (lvPPA) and nonfluent/agrammatic (naPPA) variants. Recent findings highlight that episodic memory deficits appear to discriminate these PPA variants from each other, as only lvPPA perform poorly on these tasks while having underlying amyloid pathology similar to that seen in amnestic dementias like Alzheimer's disease (AD). Most memory tests are, however, language based and thus potentially confounded by the prevalent language deficits in PPA. The current study investigated this issue across PPA variants by contrasting verbal and non-verbal episodic memory measures while controlling for their performance on a language subtest of a general cognitive screen. A total of 203 participants were included (25 lvPPA; 29 naPPA; 59 AD; 90 controls) and underwent extensive verbal and non-verbal episodic memory testing, with a subset of patients (n = 45) with confirmed amyloid profiles as assessed by Pittsburgh Compound B and PET. The most powerful discriminator between naPPA and lvPPA patients was a non-verbal recall measure (Rey Complex Figure delayed recall), with 81% of PPA patients classified correctly at presentation. Importantly, AD and lvPPA patients performed comparably on this measure, further highlighting the importance of underlying amyloid pathology in episodic memory profiles. The findings demonstrate that non-verbal recall emerges as the best discriminator of lvPPA and naPPA when controlling for language deficits in high load amyloid PPA cases.