I think the flavor of these gems (and set items) idea is
great.
The rules for it, as usual, are unnecessary... Do you really need a new mechanic to tell you that adding a skull-shaped amethyst gives your weapon the ghost touch property? Why not just making the flavor part of the core rule for adding this property to a weapon?
New mechanics always introduce possible problems, loopholes etc. These gems have a fixed cost, so it's obvious that they don't follow the normal cost progression of D&D magic items, which is normally geometrical. The limit of 1 per item at least doesn't make it too powerful, but with the added (unprecedented) feature that you can remove a gem later when you're bored, there is no reason* not to get a certain property via a gem rather than in the core way. What else is this if not a little power creep?
*other than limited availability of gems, except that many gamers as usual will demand to find them on sale "in any city big enough"