Microglia rapidly respond to changes in neural activity and inflammation to regulate synaptic connectivity. The extracellular signals, particularly neuron-derived molecules, that drive these microglial functions at synapses remain a key open question. Here we show that whisker lesioning, known to dampen cortical activity, induces microglia-mediated synapse elimination. This synapse elimination is dependent on signaling by CX3CR1, the receptor for microglial fractalkine (also known as CXCL1), but not complement receptor 3. Furthermore, mice deficient in CX3CL1 have profound defects in synapse elimination. Single-cell RNA sequencing revealed that Cx3cl1 is derived from cortical neurons, and ADAM10, a metalloprotease that cleaves CX3CL1 into a secreted form, is upregulated specifically in layer IV neurons and in microglia following whisker lesioning. Finally, inhibition of ADAM10 phenocopies Cx3cr1(-/-) and Cx3cl1(-/-) synapse elimination defects. Together, these results identify neuron-to-microglia signaling necessary for cortical synaptic remodeling and reveal that context-dependent immune mechanisms are utilized to remodel synapses in the mammalian brain.