I mean, there is obviously two components. First is avoiding any appearance of impropriety, regardless of how Pence feels. He's basically saying to his wife (and the American public by now), I will give you zero reason to believe I am being unfaithful. Second, you don't have to infer he's concerned about ripping off his clothes when he's alone with another female. Affairs in real life don't work like that. They start with building a personal relationship and then increasingly confiding/building an intimate relationship with another person rather than your spouse.
It is also funny how several people have rushed to judge Pence for objectifying women, when they don't even consider that maybe Pence's concern is that women might try to jump in the pants of a silver fox like Pence and he does not want to tempt them. So maybe you should just stop and think about who is actually viewing women as purely sexual objects here.