I'd break it down to this. Who wins:
Two untrained contestants? The side with (more) natural talent.
One side with proficiency? The side with proficiency.
Two proficient contestants? The side with (more) natural talent.
...
Secondary point: consider combat and non-combat cases separately in your schema to rewrite how bonuses are used.
Yes, I've been looking at combat and non-combat cases, but hopeful that a consistent approach will work across the board.
While I'm comparing to RAW, I use a slightly different rule for critical hits that incorporates the concept of "confirming" your critical hit. Instead of rolling a die, though, your attack simply has to be 5 or more than what was needed to hit.
The purpose of this is to eliminate the fact that when a creature can only hit on a 20, then every hit is a critical hit.
But I think that this works well regardless of that, but I'll expand.
First, my current concept is this:
Untrained - add your ability modifier (maximum +1)
Proficiency - proficiency modifier or ability modifier, whichever if higher. In most cases, this means you'll have a +1 advantage over other proficient characters for your prime requisite.
Expert - proficiency plus ability modifier. This puts a spread of up to 3 points for experts (since it's pointless to have expertise if all you'll get is a +1 bonus).
So at 1st level (assuming standard array):
Untrained: +1 maximum.
Proficient: +3 for prime requisite, +2 for everything else, except the 8 which would be +1.
Expert: +5 for prime requisite, +4 for second highest, +3 for the next two, down to +1 for lowest score.
Okay many professional athletes will tell you talent (attribute bonus) can only take you so far then skill must take over so to perhaps get this concept to work as you are suggesting why not do this.
Raw Attribute Bonus (RAB) << always round up -- necessary to reward someone for that max 20 attribute
Proficiency Bonus (PB) << always round down -- keeps the RAW + Skill from getting to high
No Proficiency = RAB
Proficiency..... = RAB + PB/2 (until PB >= RAB then PB + RAB/2)
.......This way folks with a lot of Talent can still outclass a person with no Talent and training but only up to a point.
+5 Max on Raw Talent
+6 Max on Pure Skill
+9 Max on Skill + Talent
Okay that looks pretty good but as you pointed out Expertise at x2 is a bit much and while 1.5 seems viable it might not be quite enough with an Extreme Max of +12 at level 20
So if you feel that is true then instead of having Expertise mess with the PB directly have it mess with the PB increment value which is currently = 2 + ([Level - 1] mod 4)
Which equates out to about 2 + (1 for each 4 levels)
So we decrease this for Expertise such that = 2 + ([Level - 1] mod 2) which also gives a smoother increase pattern
Which equates out to about 2 + (1 for each 2 levels)
The new maximum Proficiency Bonus is now +11 (at level 20) plus Max Attribute Bonus of +3 (at 20) makes the absolute Extreme Maximum Skill Bonus +14 at Level 20 with a 20 Attribute
This seems fine not too high but not to low it allows you to make a Nearly Impossible roll DC 30 with a 25% chance of success at Extreme Max Skill Bonus
Maybe my math is off, but I'm getting a maximum proficiency bonus with expertise as +11 (+6 for the proficiency bonus and +5 for your ability modifier). I'm not sure what you mean by Max Attribute Bonus.
So no rounding needed just:
Untrained: Ability modifier, maximum of +1 (Maximum of +1 obviously)
Proficiency: The higher of your proficiency bonus or ability modifier (Maximum of +6, since it will exceed your ability modifier)
Expertise: Proficiency bonus + ability bonus. (Maximum of +11)
A little more food for thought, should character level play into untrained?
Instead of:
Untrained = Ability modifier, maximum of +1; it could be:
Untrained = Ability modifier, maximum of proficiency bonus -1.
That way higher level characters get to use more of their ability modifier, showing some experience in similar situations, if not actual training.