There is increasing evidence that the risk of SARS-CoV-2 infection among vaccinated individuals is variant-specific, suggesting that protective immunity against SARS-CoV-2 may differ by variant. We enrolled vaccinated (n = 39) and unvaccinated (n = 11) individuals with acute, symptomatic SARS-CoV-2 Delta or Omicron infection and performed SARS-CoV-2 viral load quantification, whole-genome sequencing, and variant-specific antibody characterization at the time of acute illness and convalescence. Viral load at the time of infection was inversely correlated with antibody binding and neutralizing antibody responses. Increases in antibody titers and neutralizing activity occurred at convalescence in a variant-specific manner. Across all variants tested, convalescent neutralization titers in unvaccinated individuals were markedly lower than in vaccinated individuals. For individuals infected with the Delta variant, neutralizing antibody responses were weakest against BA.2, whereas infection with Omicron BA.1 variant generated a broader response against all tested variants, including BA.2.