Viral infections have variable outcomes, with severe disease occurring in only few individuals. We hypothesized that this variable outcome could correlate with the nature of responses made to previous microbes. To test this, mice were infected initially with influenza A virus (IAV) and in memory phase challenged with lymphocytic choriomeningitis virus (LCMV), which we show in this study to have relatively minor cross-reactivity with IAV. The outcome in genetically identical mice varied from mild pneumonitis to severe acute lung injury with extensive pneumonia and bronchiolization, similar to that observed in patients who died of the 1918 H1N1 pandemic. Lesion expression did not correlate with virus titers. Instead, disease severity directly correlated with and was predicted by the frequency of IAV-PB1703- and IAV-PA224-specific responses, which cross-reacted with LCMV-GP34 and LCMV-GP276, respectively. Eradication or functional ablation of these pathogenic memory T cell populations, using mutant-viral strains, peptide-based tolerization strategies, or short-term anti-IFN-gamma treatment, inhibited severe lesions such as bronchiolization from occurring. Heterologous immunity can shape outcome of infections and likely individual responses to vaccination, and can be manipulated to treat or prevent severe pathology.