As the more experienced and focused segment of rpg players out there, how much modularity do you feel is too much?
It depends on who you are referring to. How much is too much for a company of designers? That will be determined a great deal by demand and the ability to supply. In general, more product and variety of product is a positive. Seeing as these products are digital primarily, but can also be in printed text and/or game piece / board product forms. Design costs and quality are also important. Release too much bad product and your other products will be soured as well. Release only enough to keep your quality high and you may be leaving some of the market to 3rd party publishers, if you allow outside companies to create modules too.
In terms of players and playing groups: again, it depends upon the people in question. I've known people who really only prefer "light" or small games with few rules and low preparation - think Cheap Ass Games. OTOH there are more than a few buyers during the 3.x boom who literally not only owned dozens to 100s of books, but allowed every possible text available as supplemental rules material for their games. Some want low, some want high. It's going to depend upon the customer.
As a business strategy you may want to release a small core game as you say and then supplement it with convertible modular systems. That means you can run everything in the core game, but for those who want mass combat you have a mass combat add-on. Those who want dueling get the dueling set. Those who want naval warfare buy that set. I strongly suggest each of these not only be contiguous with each other via the core game mechanics / stats, but also that they must be able to stand alone as quality single play games. That means you can buy that board game, play it as is, and use it to supplement an ongoing RPG campaign. But keep these optional. If they don't want two hour long skirmish combat rules, then don't put them in the core books.
What qualifies as core, both statistically and via die roll (or other) resolution is central to allowing you to not only have an attractive game, but also defines (therefore limiting) you design space for following modules.
Also, if you're looking at selling adventure modules, something different than supplemental systems for different focuses within the game, you may want to look at ASL scenarios as a template. It has a core game, additional rules, and publishes many scenarios as preplanned war sites. By doing so you not only sell a product to relieve DMs of heavy planning with (by their option) heavy module campaigns, but also are tying together and making meaningful tile sets and other peripherals which end up being simply blocks of wood or artful cardboard otherwise.
At what point do system modules become a liability or cause a lack of focus?
When a new buyer cannot figure out what to buy you have a liability. The core game needs to be recognizably "the one to always buy first" out of any of your particular game's product line. The ability to stand alone for the others is a good strategy IMO, but the understanding is, according to contemporary game selling, additional products are attempts at self supporting advertisement to the core products. They are what makes the majority of profits. Well, at least that's how I take it from what Dancey has said.
Is there a threshold for when too many game modules are in print? I guess for the publisher it comes into what the rate of return is. Are you still gaining sales? Also, how relevant is the particular focus of the module to the overall game? Basket weaving the Module(tm) is unlikely to sell due to low relevance. For customers modules are about how deep in terms of complexity they want to go for their game in a particular focus. An underwater campaign is going to be using 3D combat and underwater travel systems far more often than one focused on desert life and pyramid/tomb exploration. Focus is determined by relevance to each group, player, and each particular campaign they are in.
The X-factors here are quality game design and overall cost for a hobbyist. If you design "Ticket to Ride" as a modular supplement as well as a stand alone game, you may have a winner on your hands. Will developing railroad lines be relevant to every campaign? It depends on the module's theme really. The point is that a supplement could become a successful game in its own right as a standalone and allowing designers enough game space to create such things is part of what goes into making a good core game. In terms of cost by hobby gamer it comes down to how many, say, board game boxes as module type they are willing to buy. The economy isn't going to necessarily support dozens for every group. Higher price items also shrink sales while not necessarily demand because the product is simply out of the price range for the customer base you are selling to. OTOH a $10,000 game doesn't need to sell many units to be profitable either. It comes down to good business and market sense. In the end, most of what is being sold is squiggled ink on paper, something we can all get online for free. It's up to the publishers to create such products that make owning physical copies or subscription services to online continuous releases a desirable purchase.