If a PC wants the advantages, I think they have to take the disadvantages too. In practice, I haven’t seen very many PCs clamour for either.
NPC A has a neat combat trick that she used against the PCs. She spent 10 years learning this trick from master X in city Y. If a PC wants to learn the trick, they would have to leave the party for 10 years.
NPC B can cast spells without using components. She jammed a necrotic shard into her mind and went insane. Does any of the PCs want to do the same (including the hoing insane part)?
NPC C also has a cool combat trick. He sacrificed a lot to get it, as evidenced by the fact that his statblock indicates he is trained in 2 skills and has no feats. Are any of the PCs willing to give up three skills and all feats for this trick?