I think Books 1 & 2 are unequivocal classics and there is an innocence and magic to them unmatched by the rest of the series. Prisoner of Azkaban took an unexpected dark turn but the time turner eventually went from something cool to the most blatant use of Deus Ex machina in fantasies modern history.

Goblet of Fire was the first time the world expanded and I think the first time JK Rowlings world building really came into its own. It had the wonder of the first two books and the darkness of the third. Beautiful book.