Several general principles of global 3D genome organization have recently been established, including non-random positioning of chromosomes and genes in the cell nucleus, distinct chromatin compartments, and topologically associating domains (TADs). However, the extent and nature of cell-to-cell and cell-intrinsic variability in genome architecture are still poorly characterized. Here, we systematically probe heterogeneity in genome organization. High-throughput optical mapping of several hundred intra-chromosomal interactions in individual human fibroblasts demonstrates low association frequencies, which are determined by genomic distance, higher-order chromatin architecture, and chromatin environment. The structure of TADs is variable between individual cells, and inter-TAD associations are common. Furthermore, single-cell analysis reveals independent behavior of individual alleles in single nuclei. Our observations reveal extensive variability and heterogeneity in genome organization at the level of individual alleles and demonstrate the coexistence of a broad spectrum of genome configurations in a cell population.