Semantic deficits in Alzheimer’s disease have been widely documented, but little is known about the integrity of semantic memory in the prodromal stage of the illness. The aims of the present study were to: (i) investigate naming abilities and semantic memory in amnestic mild cognitive impairment (aMCI), early Alzheimer’s disease (AD) compared to healthy older subjects; (ii) investigate the association between naming and semantic knowledge in aMCI and AD; (iii) examine if the semantic impairment was present in different modalities; and (iv) study the relationship between semantic performance and grey matter volume using voxel-based morphometry. Results indicate that both naming and semantic knowledge of objects and famous people were impaired in aMCI and early AD groups, when compared to healthy age- and education-matched controls. Item-by-item analyses showed that anomia in aMCI and early AD was significantly associated with underlying semantic knowledge of famous people but not with semantic knowledge of objects. Moreover, semantic knowledge of the same concepts was impaired in both the visual and the verbal modalities. Finally, voxel-based morphometry analyses revealed that semantic impairment in aMCI and AD was associated with cortical atrophy in the anterior temporal lobe (ATL) region as well as in the inferior prefrontal cortex (IPC), some of the key regions of the semantic cognition network. These findings suggest that the semantic impairment in aMCI may result from a breakdown of semantic knowledge of famous people and objects, combined with difficulties in the selection, manipulation and retrieval of this knowledge.