On average, men begin to experience a decline in fertility starting in their late 30s and become significantly less fertile by their mid-40s to early 50s. Though men can still father children well into their later years, the chances of infertility and genetic abnormalities increase with age.