Well that's the idea.
1 magic but many branches of it.
There would be no separate white mages and black mages. There is a wizard who specializes in white, black, grey, blue, red, green, yellow, orange, purple, or brown magic but has access to them all.
Then you get another class that only gets access to 3 colors but specializes in all 3. You choose brown, white, and green for your druid. White, blue, and yellow for your holy cleric. etc.
Then you have you trickster class, your warlocky class, and your gish class that only gets one color but gets other stuff.