Environmental variables are often the primary drivers of species' distributions as they define their niche. However, individuals, or groups of individuals, may sometimes adopt a limited range within this larger suitable habitat as a result of social and cultural processes. This is the case for Eastern Caribbean sperm whales. While environmental variables are reasonably successful in describing the general distribution of sperm whales in the region, individuals from different cultural groups have distinct distributions around the Lesser Antilles islands. Using data collected over 2 years of dedicated surveys in the Eastern Caribbean, we conducted habitat modeling and habitat suitability analyses to investigate the mechanisms responsible for such fine-scale distribution patterns. Vocal clan-specific models were dramatically more successful at predicting distribution than general species models, showing how a failure to incorporate social factors can impede accurate predictions. Habitat variation between islands did not explain vocal clan distributions, suggesting that cultural group segregation in the Eastern Caribbean sperm whale is driven by traditions of site/island fidelity (most likely maintained through conformism and homophily) rather than habitat type specialization. Our results provide evidence for the key role of cultural knowledge in shaping habitat use of sperm whales within suitable environmental conditions and highlight the importance of cultural factors in shaping sperm whale ecology. We recommend that social and cultural information be incorporated into conservation and management as culture can segregate populations on fine spatial scales in the absence of environmental variability.