I'm still having some trouble with parsing this. If we let the Spell Damage Modifier become a variably Y, and let X refer to the size of the die (from the provided list), then total spell damage for any spell is just equal to (Y + spell level)dX?
Yes, this is the idea, but you make some good points toward needing to refine it.
If I'm following this correctly, your spell damage for burning hands will increase significantly with level, since it scales with both Proficiency and max spell level. At level 11, you're getting 7 dice of damage out of a level 1 spell slot. With so much of the damage coming out of the Y term, there's relatively little benefit to up-ranking, although this does solve the problem where fireball III would deal more damage than burning hands III.
You make a good point about getting too much out of a 1st-level slot, and pointing out the proficiency scaling. Let's presume that it is ok to say that a caster gets better even at 1st-level spells later in the game, and keep proficiency. But let's also clarify the Cha cap to make upcasting important. However, doing so eliminates the Spell Damage Modifier, which is more complicated. I prefer a one-time calculation, however. But even with this scaling of a given level's slots, they remain inferior to higher level slots.
But should an upcast spell be as good as a native spell? Probably not. Otherwise you would just take one spell at 1st level and be done with it.
When you cast a spell of 1st level or higher, you roll one damage die (dX) per level of the spell. You also gain additional damage dice equal to your proficiency bonus, and you gain dice equal to your Charisma modifier, up to a maximum of half the spell's base level.
Number of Dice = Spell Level + Proficiency Bonus + Charisma modifier (max Base Spell Level/2)
Damage = Number of Dice * dX
I also assume that your max spell level refers to spell slots, rather than spells known, to avoid penalizing multi-class spellcasters.
Yes.
One suggestion is to remove the level-based cap on Charisma contributions, just because it's too complex for this edition. Honestly, I would strongly recommend that you rework the formula so that Charisma adds a static bonus rather than influencing the dice, unless you plan on making similar changes to weapon attacks. As it stands, powerful greatweapon fighters are encouraged to maximize their Dex/Accuracy rather than Strength/Power, because the attack roll is much more important than the damage roll when you're already rolling large damage dice.
The problem with static bonuses is that they are quickly dwarfed by more/bigger damage dice, making the stat unimportant. You rightly point out that this is a problem with weapon fighters as it stands. In general, weapon fighters get more benefit from extra attack, which includes re-adding the stat bonus, whereas casters roll more dice for a single attack and would add the bonus only once, making it much less relevant. I'll have to think about this more on how to balance it.
I would also recommend reducing the die-size penalty for half-damage, since half of a d4 is a trivial amount of damage in any situation. Just take "half damage on a save" to be a special effect, like imposing a condition, so it only decreases the die-size by one step.
Dropping only a single die size makes the average damage of save-for-half spells much higher than the others. Dropping two die sizes makes them comparable. Half of a d4 is negligible, but when there are many d4s, it makes a difference.
Balancing Spell Damage
The easiest way to balance spell damage is to roll dice equal to twice the spell's level. This gives a nice, even approach. However, it does not leave room for Cha impact, and it also makes all upcasting equivalent to higher spells, making taking more than one spell pointless.
Another way:
Spell dice = Spell level + 2*Cha (max base spell level).
This takes out proficiency, making it a little bit easier, and it prevents scaling, and Charisma is important.