The anterior temporal lobes (ATLs) have been consistently associated with semantic processing which, in turn, has a key role in reading aloud single words. This study aimed to investigate (1) the reading abilities in patients with the semantic variant of primary progressive aphasia (svPPA), and (2) the relationship between gray matter (GM) volume of the left ATL and word reading performance using voxel-based morphometry (VBM). Three groups of participants (svPPA, Alzheimer’s Disease, AD and healthy elderly adults) performed a reading task with exception words, regular words and pseudowords, along with a structural magnetic resonance imaging scan. For exception words, the svPPA group had a lower accuracy and a greater number of regularization errors as compared to the control groups of healthy participants and AD patients. Similarly, for regular words, svPPA patients had a lower accuracy in comparison with AD patients, and a greater number of errors related to complex orthography-to-phonology mappings (OPM) in comparison to both control groups. VBM analyses revealed that GM volume of the left ATL was associated with the number of regularization errors. Also, GM volume of the left lateral ATL was associated with the number of errors with complex OPM during regular word reading. Our results suggest that the left ATL might play a role in the reading of exception words, in accordance with its role in semantic processing. Results further support the involvement of the left lateral ATL in combinatorial processes, including the integration of semantic and phonological information, for both exception and regular words.