I think part of the problem, Nemm, was that
Prisoner of Azkaban was a bit too ambitious with the plot. In the books, there are a lot of little side plots happening that are interesting, but only superficially important to the main plot. In the movie, they tried really hard to include everything, but due to the length of a typical movie, they only had so much time to spend on each subject. In order to make room for all those little exposition scenes, they had to dig out a few big gaping holes in the plot. The result was a script that was scattered all over the place plot-wise and made about as much sense to people who haven't read the book as David Lynch's original big screen adaptation of
Dune.
I'd recommend watching the later movies, if you haven't... They seem to have realized their mistake, and really tightened up the scripts and plots. It's part of the reason why, as Donovan mentions above, the Dobby never shows up in
Goblet of Fire and you don't see Ron on the quidditch team in
Order of the Phoenix. While interesting in of themselves and perhaps character building when taken in their entirety, they aren't especially necessary to the overall plot and only serve as a useless time-consuming distraction if they can't be fully detailed.