Neuroimaging studies on healthy subjects have reported the involvement of the bilateral anterior temporal lobes (ATL) in recognizing known people. While the role of the right ATL in the semantic processing of known people is accepted, it is still matter of debate whether left ATL is implicated in the name retrieval or in the access to semantic information. In the present fMRI study, we modulated the level of specificity of the semantic categorization task of famous face stimuli in order to map the brain regions involved in the specific semantic processing controlling for automatic name access. Specifically, participants (n = 12, mean age = 23.0+/-1.7 years, F/M = 8⁄4) were asked to determine whether the stimulus photograph matched with the label of the profession category. Since the same set of stimuli was used for both specific and general conditions, contrasting these two conditions controlled for the activation due to the automatic access to the name of famous persons. Results showed that: 1) the right ATL was activated by a semantic categorization task of famous face stimuli at both general and specific level; 2) while controlling for the activation due to the automatic name retrieval, the left ATL showed increased activation during the specific categorization task; 3) the specificity of the semantic processing increased strength of connections towards the left ATL from the right lateral fusiform gyrus, devoted to the visual processing of faces, and from the right ATL. These findings support the idea of a crucial role of the left anterior temporal lobe in the processing of person-specific semantic information.