I thought Joss did a pretty good job of explaining where everybody was coming from. Sure, having seen the show helps a lot and certain things will mean more to you if you have seen it, but I don't think you'll be lost. There are a few minor things that might need some explaination to make you see the full picture of what happened, but like I said, I don't think you'll be lost without it. Not like with people who hadn't read Prisoner of Azkaban.