Traveling back in time is an interesting topic. Neglecting the actual physical possibility or impossibility of time travel, it may actually do a pretty good job of governing itself from ever happening. For instance, suppose it is possible and there's a person with the capability of making a "time machine" or whatever and the audacity to go back with little disregard to what it does to the dimension that he's currently in. Well, if he goes back very far, there's a real possibility that he makes himself never exist ala butterfly effect. So, go back to the original time line when this certifiable genius got the idea of how to actually time travel. He then set in motion the events that would lead him to going back in time and thus ruining his chance at being born. So IN THAT TIME DIMENSION, he would simply go *poof* at the instant he thought about building the time machine. So in theory, there could have been many men that figured out time travel, or rather, would have figured out time travel but deleted their existence by deciding they were gonna do it.
Of course, this neglects the chicken or the egg argument giving you a perpetual loop in the time line which really only leaves the possibility of different dimensions. In short, if you're really smart and devious, it might be a good idea to never even consider creating a way to travel back in time.