Behavioral ecologists, evolutionary biologists, and anthropologists have been long fascinated by the existence of “monogamy” in the animal kingdom. Multiple studies have explored the factors underlying its evolution and maintenance, sometimes with contradicting and contentious conclusions. These studies have been plagued by a persistent use of fuzzy terminology that often leads to researchers comparing “apples with oranges” (e.g., comparing a grouping pattern or social organization with a sexual or genetic mating system). In this review, we provide an overview of research on “monogamy” in mammals generally and primates in particular, and we discuss a number of problems that complicate comparative attempts to understand this issue. We first highlight why the muddled terminology has hindered our understanding of both a rare social organization and a rare mating system. Then, following a short overview of the main hypotheses explaining the evolution of pair-living and sexual monogamy, we critically discuss various claims about the principal drivers of “monogamy” that have been made in several recent comparative studies. We stress the importance of using only high quality and comparable data. We then propose that a productive way to frame and dissect the different components of pair-living and sexual or genetic monogamy is by considering the behavioral and evolutionary implications of those components from the perspectives of all participants in a species' social system. In particular, we highlight the importance of integrating the perspective of “floater” individuals and considering their impacts on local operational sex ratios, competition, and variance in reproductive success across a population. We stress that pair-living need not imply a reduced importance of intra-sexual mate competition, a situation that may have implications for the sexual selection potential that have not yet been fully explored. Finally, we note that there is no reason to assume that different taxa and lineages, even within the same radiation, should follow the same pathway to or share a unifying evolutionary explanation for “monogamy”. The study of the evolution of pair-living, sexual monogamy, and genetic monogamy remains a challenging and exciting area of research.