Pretty sure you're supposed to think he was warged into old Hodor at the time, which kind of explains the eyes as well Wylis being able to hear.
Which brings me to the other thing I hate about this whole time travel situation. Bran warged into Hodor because Hodor was basically incapacitated out of fear (presumably because of the whole "hold the door" incident), but he wouldn't have been that way if Bran hadn't warged into him, which Bran wouldn't have had to do if Hodor hadn't been that way.
But I'm not that pissed about the scene itself because I thought it was beautifully done.