Transcriptional co-activator Prdm16 controls brown fat development and white fat browning, but how this thermogenic function is modulated post-translationally is poorly understood. Here, we report that Cbx4, a Polycomb group protein, is a SUMO E3 ligase for Prdm16 and that Cbx4-mediated sumoylation of Prdm16 is required for thermogenic gene expression. Cbx4 expression is enriched in brown fat and is induced in adipose tissue by acute cold exposure. Sumoylation of Prdm16 at lysine 917 by Cbx4 blocks its ubiquitination-mediated degradation, thereby augmenting its stability and thermogenic function. Moreover, this sumoylation event primes Prdm16 to be further stabilized by methyltransferase Ehmt1. Heterozygous Cbx4-knockout mice develop metabolic phenotypes resembling those of Prdm16-knockout mice. Furthermore, fat-specific Cbx4 knockdown and overexpression produce remarkable, opposite effects on white fat remodeling. Our results identify a modifying enzyme for Prdm16, and they demonstrate a central role of Cbx4 in the control of Prdm16 stability and white fat browning.