Except it always will as long as there are humans voting. It isn't unfair because the same rules pretty much apply to everyone - start with a high preseason ranking, win your games, and you'll be in much better shape than the team that started the season unranked and had similar wins.
Yea, it's a sneaky thing that people don't realize, but if you (and teams in your conference) start out highly ranked, they're probably going to stay that way. If #5 OU loses to unranked KSU early in the season, OU probably falls to #20 and KSU -might- be ranked the next week.
If #5 OU loses to #10 KSU, they probably flip spots, which gives more teams in the conference an opportunity for top-10 wins and fewer opportunities for bad losses.
What Baylor's schedule was flipped in 2012 (win all their games leading up to KSU and lose the rest)? It's these kind of arbitrary things that actually make a difference.