Over the past few years, numerous anti-phage defense systems have been discovered in bacteria. While the mechanism of defense for some of these systems is understood, a major unanswered question is how these systems sense phage infection. To systematically address this question, we isolated 177 phage mutants that escape 15 different defense systems. In many cases, these escaper phages were mutated in the gene sensed by the defense system, enabling us to map the phage determinants that confer sensitivity to bacterial immunity. Our data identify specificity determinants of diverse retron systems and reveal phage-encoded triggers for multiple abortive infection systems. We find general themes in phage sensing and demonstrate that mechanistically diverse systems have converged to sense either the core replication machinery of the phage, phage structural components, or host takeover mechanisms. Combining our data with previous findings, we formulate key principles on how bacterial immune systems sense phage invaders.