Adipose tissue development is poorly understood. Here we use a lineage-tracing strategy optimal for adipocytes to provide evidence that Myf5 precursors are not the exclusive source of brown adipocytes and contribute more to the mature white and brite adipocyte populations than previously thought. Moreover, Myf5-lineage distribution in adipose tissue changes in response to modifiable and non-modifiable factors. We also find that the Pax3 lineage largely overlaps with the Myf5 lineage in brown fat and subcutaneous white fat, but exhibits gender-linked divergence in visceral white fat while the MyoD1 lineage does not give rise to any adipocytes. Finally, by deleting insulin receptor beta in the Myf5 lineage, we provide in vivo evidence that the insulin receptor is essential for adipogenesis and that adipocyte lineages have plasticity. These data establish a conceptual framework for adipose tissue development and could explain body fat patterning variations in healthy and lipodystrophic or obese humans.