The AD&D DMG (published 1979) has the following entry on p 69:
Number Of Opponents Per Figure:
Physical size and space will dictate limitation upon the number of opponents able to engage a single figure in melee. If Official ADVANCED DUNGEONS 8 DRAGONS miniature figures are used to represent the creatures involved in a melee, then these miniatures will dictate the number of opponents which can be involved. Beware of using other fantasy miniature figures, as most of them are not designed to the specific scale and do not conform to the standards of the MONSTER MANUAL (or
even necessarily conform to 25mm scale!). In other cases use the following rule of thumb:
1. If the single figure is size S, 4 size M or 2 size L figures are all that can attack it under optimum conditions during any single round. 6 size S figures can attack it, however, space permitting.
2. If the single figure is size M, it can be attacked by a maximum of 8 size S opponents, 6 size M, or 4 size L.
3. If the single figure is size L, a maximum of 12 size S, 8 size M, or size L figures can engage it at one time.
Modifiers are impossible to detail fully, as they include the physical circumstances prevailing at the time. It is obvious that a figure in a narrow passage or in a corner can be attacked only from a limited front. Similarly, the shape of the single figure and the attackers will be a factor, as a giant snake has more body space subject to attack than does a giant, unless the attackers are able to fly or otherwise attack the entire vertical surface of the giant. Any long-bodied creature has more area open to attack, and wide-bodied creatures, or those which employ some natural body weaponry which requires contact with the opponent, will be more restricted with respect to the number of attackers possible, viz. how many constrictor snakes can effectively attack an opponent? Two basic grid systems are shown below, one with hexagons, one with squares. These grids demonstrate how multiple opponents engage a single figure. . . .
To determine the number of opponents which can attack the single defender, use squares or hexes of one inch per face for 25 mm scale, or you may simply view the illustrations used above and mentally visualize the situation.
The four diagrams that are provided ("diagonal" hexes, "square" hexes, diagonal squares, square squares) show how 6 attackers (for hexes) or 8 attackers (for squares) can surround a single figure.
The text makes it clear that both minis on grids and "theatre of the mind" (ie
mentally visualising the situation) are contemplated as approaches.
Rolemaster books (mid-to-late 80s) contemplated the use of hexes for combat positioning; from memory it's also a feature of DragonQuest. And I'm sure other systems in the 70s/80s features it. (Eg Classic Traveller (1977) defaults to "bands" which can be tracked on lined paper, "us[ing] cardboard markers or cast metal miniatures to represent the characters", but the rules suggest that a group may use squares or hexes if they want to have more tactical manoeuvring (Book 1, p 29).)
I never used squares or tokens until I GMed 4e D&D, but I was familiar with the idea from my first encounters with RPGs (which for me was Traveller).
EDIT: cross-posted with [MENTION=19675]Dannyalcatraz[/MENTION].