The mammalian cell nucleus displays a remarkable spatial segregation of active euchromatic from inactive heterochromatic genomic regions. In conventional nuclei, euchromatin is localized in the nuclear interior and heterochromatin at the nuclear periphery. In contrast, rod photoreceptors in nocturnal mammals have inverted nuclei, with a dense heterochromatic core and a thin euchromatic outer shell. This inverted architecture likely converts rod nuclei into microlenses to facilitate nocturnal vision, and may relate to the absence of particular proteins that tether heterochromatin to the lamina. However, both the mechanism of inversion and the role of interactions between different types of chromatin and the lamina in nuclear organization remain unknown. To elucidate this mechanism we performed Hi-C and microscopy on cells with inverted nuclei and their conventional counterparts. Strikingly, despite the inversion evident in microscopy, both types of nuclei display similar Hi-C maps. To resolve this paradox we developed a polymer model of chromosomes and found a universal mechanism that reconciles Hi-C and microscopy for both inverted and conventional nuclei. Based solely on attraction between heterochromatic regions, this mechanism is sufficient to drive phase separation of euchromatin and heterochromatin and faithfully reproduces the 3D organization of inverted nuclei. When interactions between heterochromatin and the lamina are added, the same model recreates the conventional nuclear organization. To further test our models, we eliminated lamina interactions in models of conventional nuclei and found that this triggers a spontaneous process of inversion that qualitatively reproduces the pathway of morphological changes during nuclear inversion in vivo. Together, our experiments and modeling suggest that interactions among heterochromatic regions are central to phase separation of the active and inactive genome in inverted and conventional nuclei, while interactions with the lamina are essential for building the conventional architecture from these segregated phases. Ultimately our data suggest that an inverted organization constitutes the default state of nuclear architecture.