The obesity epidemic has intensified efforts to understand the mechanisms controlling adipose tissue development. Adipose tissue is generally classified as white adipose tissue (WAT), the major energy storing tissue, or brown adipose tissue (BAT), which mediates non-shivering thermogenesis. It is hypothesized that brite adipocytes (brown in white) may represent a third adipocyte class. The recent realization that brown fat exist in adult humans suggests increasing brown fat energy expenditure could be a therapeutic strategy to combat obesity. To understand adipose tissue development, several groups are tracing the origins of mature adipocytes back to their adult precursor and embryonic ancestors. From these studies emerged a model that brown adipocytes originate from a precursor shared with skeletal muscle that expresses Myf5-Cre, while all white adipocytes originate from a Myf5-negative precursors. While this provided a rational explanation to why BAT is more metabolically favorable than WAT, recent work indicates the situation is more complex because subsets of white adipocytes also arise from Myf5-Cre expressing precursors. Lineage tracing studies further suggest that the vasculature may provide a niche supporting both brown and white adipocyte progenitors; however, the identity of the adipocyte progenitor cell is under debate. Differences in origin between adipocytes could explain metabolic heterogeneity between depots and/or influence body fat patterning particularly in lipodystrophy disorders. Here, we discuss recent insights into adipose tissue origins highlighting lineage-tracing studies in mice, how variations in metabolism or signaling between lineages could affect body fat distribution, and the questions that remain unresolved. This article is part of a Special Issue entitled: Modulation of Adipose Tissue in Health and Disease.