Sequence homology between SARS-CoV-2 and common-cold human coronaviruses (HCoVs) raises the possibility that memory responses to prior HCoV infection can affect T cell response in COVID-19. We studied T cell responses to SARS-CoV-2 and HCoVs in convalescent COVID-19 donors and identified a highly conserved SARS-CoV-2 sequence, S811-831, with overlapping epitopes presented by common MHC class II proteins HLA-DQ5 and HLA-DP4. These epitopes are recognized by low-abundance CD4 T cells from convalescent COVID-19 donors, mRNA vaccine recipients, and uninfected donors. TCR sequencing revealed a diverse repertoire with public TCRs. T cell cross-reactivity is driven by the high conservation across human and animal coronaviruses of T cell contact residues in both HLA-DQ5 and HLA-DP4 binding frames, with distinct patterns of HCoV cross-reactivity explained by MHC class II binding preferences and substitutions at secondary TCR contact sites. These data highlight S811-831 as a highly conserved CD4 T cell epitope broadly recognized across human populations.