Appeal to authority? I am mostly known around here for physics, but I am by no means ignorant of software development projects.
The rules as written are not rocket science. But making the system extensible gets hairy, unless you want a development effort any time you add supplemental rules.
And the nigh-arbitrary house rules? Those are a nightmares. "I have a ring that gives me a +6 to Dex, but only when flying...". And every campaign is going to have some of those weird things the GM made up, and doesn't fit. And each time you shove folks out into "just remember to do that yourself" is a place where you limit how good your virtual tabletop (which, of course, should be compatible) can be.
As someone noted upthread, houserules almost call for something like a scripting language. That's not easy to do, and it is less easy to do for people who don't write computer programs for a living.
UX, at the level we are talking about, if we get it wrong, the product doesn't sell and we are down something close to a million dollars.
Actually, it probably wouldn't be all that bad.
What we're talking about here is: A series of states, a series of variables, and a series of rules that affect states or variables. A scripting language is probably the easiest way to go about implementing it, but by no means the only way.
Your example of the right is a good one. "I have a ring that gives me a +6 to Dex, but only when flying", broken down to important keywords it is "Ring gives +6 to Dex when flying", written out into a "Rule" it is "[object] [increases] [target variable] [value] [in state]" A pretty substantial portion of the game can be expressed that way, and can be even further generalized to make it even more flexible..."[source] [increases] [target variable] [value] [in state]" and it gives us a "rule" which can express any positive magic item, spell, or special ability and we can just use the same thing to decrease or just change [increase] to [alters] and let the +/- handle the effect.
At its root, as far as any VTT goes, all you're doing is some fairly basic math, you never actually transform anything. "State" in this sense is just some value attached to the character/critter.
That's never been a VTT's problem really, a VTT's problem's are...
1. The game allows arbitrary input and while the character can easily be modeled, the events of a game can never be. I can model how the citizens in a town would react to various reputation states, but I can never model the Player's ability to arbitrarily state "I pee in the town square's fountain" and have the VTT compute the results.
2. The VTT requires a pretty substantial amount of graphics to be useful, I can describe anything in an RPG, a VTT needs to be able to display an enourmous range or it's really just a giant automated spreadsheet.
Modeling the gamerules isn't the hard part, handling the input to the system and not showing the same image for every tiles/critter/etc is the hard part. Honestly, the best use of a VTT is probably just as a battleboard and the aforementioned spreadsheet and let the rest of the game play as it always did. Trying to automate a game that allows arbitrary input and infinite environments/appearances is never going to work.