Drugs that inhibit HIV-1 replication and prevent progression to AIDS do not eliminate HIV-1 proviruses from the chromosomes of long-lived CD4+ memory T cells. To escape eradication by these antiviral drugs, or by the host immune system, HIV-1 exploits poorly defined host factors that silence provirus transcription. These same factors, though, must be overcome by all retroviruses, including HIV-1 and other primate immunodeficiency viruses, in order to activate provirus transcription and produce new virus. Here we show that Vpx and Vpr, proteins from a wide range of primate immunodeficiency viruses, activate provirus transcription in human CD4+ T cells. Provirus activation required the DCAF1 adaptor that links Vpx and Vpr to the CUL4A/B ubiquitin ligase complex, but did not require degradation of SAMHD1, a well-characterized target of Vpx and Vpr. A loss-of-function screen for transcription silencing factors that mimic the effect of Vpx on provirus silencing identified all components of the Human Silencing Hub (HUSH) complex, FAM208A (TASOR/RAP140), MPHOSPH8 (MPP8), PPHLN1 (PERIPHILIN), and MORC2. Vpx associated with the HUSH complex components and decreased steady-state levels of these proteins in a DCAF-dependent manner. Finally, vpx and FAM208A knockdown accelerated HIV-1 and SIVMAC replication kinetics in CD4+ T cells to a similar extent, and HIV-2 replication required either vpx or FAM208A disruption. These results demonstrate that the HUSH complex restricts transcription of primate immunodeficiency viruses and thereby contributes to provirus latency. To counteract this restriction and activate provirus expression, primate immunodeficiency viruses encode Vpx and Vpr proteins that degrade HUSH complex components.