OBJECTIVE: To investigate the neural correlates of verbal and nonverbal semantic processing in neurodegenerative disease. BACKGROUND: Semantic memory is often impaired in neurodegenerative disease. Neuropsychologic and functional neuroimaging studies suggest that the semantic processing of verbal and nonverbal stimuli may depend on partially distinct brain networks. METHODS: We examined this possibility using voxel-based morphometry to correlate performance on verbal and nonverbal versions of a semantic association task with regional gray matter atrophy in 144 individuals with a variety of neurodegenerative diseases. RESULTS: Results showed that, regardless of stimulus type, semantic processing correlated with atrophy in both temporal lobes. In addition, material-specific correlations were found in left temporal regions for verbal stimuli and the right fusiform gyrus for nonverbal stimuli. CONCLUSIONS: These results provide evidence for a differential role of the left and right hemispheres in the extraction of semantic information from verbal and pictorial representations. Areas in right inferior temporal lobe may be necessary to access structural descriptions of visually presented objects.