I don't know how many fights you have been in versus how many movies you have seen, but in my experience the most dangerous people do not have an "ape index" (whatever that is). People swagger and saunter all the time that have no real skill (more often than not, in fact) and ones that can fight often do not flex until it happens.
In other words: it is pure fantasy to suggest that in the real world people walk around with a hit point bar and level hovering above their heads. Acting like that is a real thing is something people who don't know what they are talking about do.
The most effective fencers I've known don't swagger about; the next tier down often do. Then, down a rung further are folks like me who make a good show, but know better than to swagger because it invites actual attention and caution from groups 1 and 2...
But could they observe exactly how bad a fighter you are?
Only if that lack of skill is due to physical limitations. For example, a person with Cerebral Palsey is going to suck at fencing - tho' I'm certain one such person would love to try...
Skilled swordsmen do tend to have a few visible traits... specific callouses, certain muscles not to scale, often some impressive bruises if in tier 2 or 3...
... but most of them aren't readily apparent from a distance. It's not like the English Longbowmen whose handedness can be told from bilateral dimorphism.
Given enough practice, the swordsman's muscles' attachments will become broader on the sword arm. The callouses on the hands tend to be different by style of warfare - thrusting shortswords get face of palm and side of index and palm. Broadsword, when I studied it, I got index and pinky on the palm-side, and index on the inside between the distal and proximal knuckles - from hilt rub.
Fencing, when I practiced a lot and wore thin gloves, had thin callouses — able to be felt but not visible — on all faces but the backside between the distal and proximal interphallangeal knuckles, and on the index-side of the middle finger in the same distances, plus the pinky, between distal and proximal on palm, outside, and back... all from hilt contact causing rubbing. when I switched to thicker gloves, those reduced to nothing, save the right index left side, proximal to distal, since that's also used on the violin and viola on the bow.
The moment the sword comes up, more can be told. The Salute will, if you can see the arm, give away a lot... but it's a funny thing — few fighters fight with bare arms. ANd those who do tend to be DAMNED good. Or not taking safety into account. (I used to practice without armor using fiberglass rapier with a friend, because as long as we didn't aim for the head, no major injuries would ensue... and neither of us were novices with swords.) What it gives away: are they clenching the hilt? Or just gently holding it. Are they maintaining smooth tip movement or not? Are they in trigger-finger on a rapier or cruciform sword?
All of that? If you can see it unaided, you're probably in attack range. (Attack range from en garde with a 42" rapier is almost 3m body to body.)
If they're en garde, and you're undrawn, two steps and a fleché can be 5m (again, Body to body)... probably before you can clear the sheath.