Yield loss of crop plants to pests, pathogens, and competitors has remained largely undiminished over the last 50 years, representing a significant cost to farmers and a continuing threat to food security. We review the contribution of pathogens and predators to seed losses, distinguishing events that occur before and after seed dispersal. We include both weeds and crop plants because the greatest yield improvements can often result from targeting weed seed survival after crops are harvested. We highlight the strong potential for management practices to enhance predation losses of weed seeds relative to seed pathogen losses, particularly for species with long-lived dormant seeds. Many weed species have coexisted with crop plants for centuries. Studies of the chemical and physical defenses of weed seeds, and of the microbial associates that underpin their success, may also inform new approaches to manage crop seed losses.