The problem is that modern people would be horrible at the survival type stuff that would be most urgent. It would be like "Hey, can we eat these berries?" And the other guy would be like "I dunno man I usually eat Chipotle" and probably 80k would die within 5 years. Then, the 20k remaining would take a long ass time to even like, domesticate a horse and grow an ear of corn.
We're talking 2500 years MINIMUM.