Perceiving and recognizing objects through vision is a major part of most people’s lives. The neural mechanisms underlying these complex functions are still being researched. One avenue to explore this is through cases of visual object agnosia: a deficit in object recognition despite intact vision. In this systematic review, we compiled the findings of seven studies, all using fMRI to study individuals with visual object agnosia. Visual agnosia has been investigated for over 130 years and the history of it is presented. We discuss and critique diagnostic terminology of disorders of visual object recognition. Visual agnosia can present with individual differences that call these terms, some of which have remained unchanged since the early days of the field, into question. Our results indicate a neural network spread across brain areas in both ventral and dorsal visual streams in both hemispheres, working together to make visual object recognition happen. We found that a unilateral lesion is sufficient to cause widespread changes in the activation of distal brain regions. The articles included provide new information about what information is required for the recognition of objects, including more than just shape perception, which has been the main interest of older studies. Studying visual object agnosia thus provides interesting insights into the functioning of object recognition in the healthy human brain.