A background investigator failed to check a would-be trooper’s mental health history, allowing him to be hired for the Virginia State Police the year before he sexually extorted and kidnapped a 15-year-old girl and killed three members of her family in California, officials said.