The accessory protein Nef of human immunodeficiency virus type 1 (HIV-1) has long been known to enhance the infectivity of HIV-1 progeny virions. The multipass transmembrane proteins serine incorporator 3 (SERINC3) and SERINC5 were recently identified as novel antiviral proteins that restrict HIV-1 infectivity. Nef enhances HIV-1 infectivity by removing SERINCs from the plasma membrane, which prevents their incorporation into progeny HIV-1 virions. To exploit this potent intrinsic antiretroviral factor for potential therapy development, it is critical to explore the determinants in SERINC5 that govern its downregulation by Nef and its restriction on HIV-1 infectivity. Here I report that the ability to inhibit HIV-1 infectivity is conserved among vertebrate SERINC5 proteins, whereas the sensitivity to downregulation by Nef is not. However, a Nef-resistant SERINC5 became Nef-sensitive when its intracellular loop 4 (ICL4) was replaced by that of Nef-sensitive human SERINC5. Conversely, human SERINC5 became resistant to Nef when its ICL4 was replaced by that of a Nef-resistant SERINC5. In general, ICL4 regions from SERINCs that exhibited resistance to a given Nef conferred resistance to the same Nef when transferred to a sensitive SERINC, and vice versa. I demonstrate that human SERINC5 can be modified to restrict HIV-1 infectivity even in the presence of Nef. Moreover, by generating chimeras between SERINC5 and SERINC2, which does not exhibit antiretroviral activity, I demonstrate that SERINC5’s inhibitory function, unlike the sensitivity to Nef, requires the participation of more than one region. Helix 4 and extracellular loop 5 (ECL5) of SERINC5 are both required for the potent restriction of HIV-1 infectivity. In contrast, a large amino-terminal portion of SERINC5 is not required for its antiretroviral activity of SERINC5. The determinants in ECL5 disperse throughout the loop. Furthermore, the ECL5 of SERINC5 is a hotspot region that determines the Env-dependent antiretroviral activity of SERINC5.