Sounds like the issue is a disconnect between the game the players want to play and the game the DM wants to run (and you seem to want the same thing as the DM).
Personally I loath taking notes. I hated it in collage (and rarely did it) and still do. So if I miss a plot point sometimes then oh...