One last thing. I believe I'm applying the rules as written. I asserted at the beginning of the thread that the heart of this debate was however that I and almost everyone else regularly applied a house rule at the table, and it was the wide use of that house rule that was causing the trouble. That house rule is, "You don't have to roll initiative at the start of an encounter."
I disagree. Having played 3e and 3.5e in at least 3 countries in Living Greyhawk(where no house rules were allowed so we argued a lot about EXACTLY what the text in the books meant) with at least 100 different players(including in groups at GenCon and with DMs and players who have been in games with members of the R&D team), I can tell you no one I know ever played it like that.
I'm not saying you can't run it like that. But I disagree you are applying the rules as written and intended.
I think it has to due with your interpretation of threat and encounter. To me, something doesn't become a "threat" until you absolutely know that they are attacking you. Someone holding a sword who might be an assassin isn't a threat until you attack them or they attack you.
An encounter begins when the first action that would begin a "combat" takes place. An encounter is the space between the beginning of combat and the end of combat. Simply talking to someone doesn't start an encounter. Otherwise we'd be rolling initiative when we enter a bar and staying in initiative the entire time there were people around. I hope we are in agreement that this was not intended.
The idea of Flatfootedness is an interesting one. One of the first arguments I ever had about 3e rules(about 2 sessions into our first game) was between a DM and I over what flatfootedness was. He said it was impossible. No one was ever caught "flatfooted" and that if you expected combat, you'd obviously be NOT flatfooted. That discussion ended up with him removing the concept from his game and Rogues being no fun to play anymore, since they got sneak attack so little it wasn't worth playing them any more(I should know...I was the Rogue when he implemented this rule. And I had Improved Initiative in order to make sure I acted first more often).
I tried to explain to him that the concept was that you were simply caught off guard because you weren't "on the balls of your feet", "in a fighting stance", "knowing where to expect the attack from", "having your weapon in the right grip", and similar concepts that model the "I'm ready, but I'm not QUITE ready" concept of flatfootedness.
If you roll initiative at the start of every scene where a threat is or might be present or where either side is on their gaurd, the whole argument goes away and if I was presented with rules lawyerish players, then that's precisely how I would handle it. We'd roll initiative at the beginning of every encounter regardless until such time as the players shut up and agreed to not try to run the game.
Then you would be completely incapable of doing what I said in the previous post. Which is my favorite image of a Rogue ever. You see it in movies all the time. The image of the "Rogue" as a guy so fast that, even though you're standing in front of him not sure what to make of him, weapon in your hand, he pulls a dagger out of his sleeve and throws it at your neck before in one lightning fast action that catches you off guard and hits you before you have a chance to defend against it. Which is so obviously a sneak attack.
Unfortunately, in your system, it means that you roll initiative when you walk into the room and see the unarmed Rogue. After all, he MIGHT be a threat, and you should roll for initiative because an encounter has started. The fighter readies his action to attack if the Rogue makes a move. He then starts asking the Rogue some questions, the Rogue taunts him. Then, with lighting fast reflexes he whips out a dagger....and dies due to the Readied Action. Sure, it's great to be the Fighter in that circumstance. It sucks to have spent feats on Quick Draw and Improved Initiative when they give you no benefits at all(or very little).
Secondly, as to the 'can you take a combat action out of combat', my interpretation of when the game is in a combat state is when there is some character NPC or PC that wants to take a combat action. There really aren't any other guidelines in the rules. If the PC offers a combat action, its not my place to say, "No, you can't do that because we are not in combat now." I'm not a DM that says, "No."
It's not about saying no. It's about keeping the game balanced. Whenever an option becomes so useful that I can't come up with a reason NOT to do it, it is too powerful. If an option is too powerful, it either needs a house rule or I may be misinterpreting the intent of the rule and running it wrong.
Readying constantly out of combat was way too useful It was one of my first house rules. I was glad it became a rule in 3.5e.
As for the Total Defense option. I ran it precisely by the rules. You can take the Total Defense option out of combat all you wanted to, but it gave you a dodge bonus to your AC. You lose all dodge bonuses when flat footed. You are flat footed from the time combat begins to when you take your first action in combat. Therefore allowing those with higher Initiative than you to hit you with sneak attacks and catch you off guard, thus removing the benefit for the Total Defense option. It also only lasts for 1 round. Which ends when you take your next action. In other words, it has no effect at all outside of combat...because EVERYONE is defending themselves all the time when they aren't attacking. That's the default position.
If you can describe something which is reasonable and which even a 5 year old can do, its my job to say 'Yes' to it, not to try to figure out how to say, 'No' That's the guiding philosophy of my game refereeing. It's not merely my job to enforce the rules. My job is turn player propositions into concrete rules. The rules say that if a player proposes to do nothing but defend themselves, they recieve a small bonus to defense in exchange for small penalties in other ways. That seems perfectly reasonable to me, so I go with it.
There's a difference between describing an action and its in game effect. The rules tell you when an action sometimes fails or has a different effect then you'd expect.
To a player that said "I'm keeping my shield up defending myself from all attacks as I walk down the hallway", I'd simply say "Alright, you keep your shield up, you are ready to defend against anything. You are looking left, right, back, forward, keeping your shield at the ready for any attack, not knowing where or when it'll come. Then, suddenly, you see an Orc walk around the corner. Roll initiative. He wins. He's fast. He runs towards you and attacks in one fluid motion. He's so fast, that although you have your shield up, he attacks downwards, at your legs. Not expecting that move, you are thrown off guard and he cuts right underneath your shield and hits you soundly in the legs. You watch his weapon, his eyes, and his stance closely now, ready to move your shield to compensate for any move he makes."
Which is a long way of saying "It's the first round of combat, you are flatfooted and lose all dodge bonuses to AC since you haven't acted yet."
And the interactions of those rules are not coincidental. There's a reason it's a dodge bonus.
It sounds like your problem is the same one that my first 3e DM had. He didn't LIKE the concept of flatfootedness.