I'm now into Storm of Swords. I know watching shows first changes my perspective, but I like the changes the show has made, for the most part. Rob/Jeyne get a much fuller story, for example. That's been my biggest surprise so far in the books, what a relatively minor role Rob gets.
Books excel at the back story, and I wish some more of that had worked its way in. The Petyr/Lysa/John Arryn thing was really interesting and better explains what little they put in the show.