HIV-1 Nef enhances virion infectivity by counteracting host restriction factor SERINC5; however, the impact of natural Nef polymorphisms on this function is largely unknown. We characterize SERINC5 downregulation activity of 91 primary HIV-1 subtype B nef alleles, including isolates from 45 elite controllers and 46 chronic progressors. Controller-derived Nef clones display lower ability to downregulate SERINC5 (median 80% activity) compared with progressor-derived clones (median 96% activity) (p = 0.0005). We identify 18 Nef polymorphisms associated with differential function, including two CTL escape mutations that contribute to lower SERINC5 downregulation: K94E, driven by HLA-B( *)08, and H116N, driven by the protective allele HLA-B( *)57. HIV-1 strains encoding Nef K94E and/or H116N display lower infectivity and replication capacity in the presence of SERINC5. Our results demonstrate that natural polymorphisms in HIV-1 Nef can impair its ability to internalize SERINC5, indicating that variation in this recently described function may contribute to differences in viral pathogenesis.