I'm not actually too sure on the why, but Prisoner of Azkaban has definitely always been my favourite book to read. Perhaps it's something to do with the way it's written so that everything is connected so well even with the time travel aspect.