You are correct; a circle of radius r centered at the origin in this geometry is a square of side length 2r+1. In 4e, this is exactly how it worked. For convenience, it uses chessboard/Chebyshev distance (the L∞ metric) and foregoes a correction for off-axis movement.
The long and short of it is that you cannot have all of the following:
A: discretized, integer-indexed space ("a grid")
B: completely covered space (all points are contained in, or at the edge of, a tile)
C: movement without imperfect/complex conversions (e.g. no "2 diagonal squares = 3 horiz/vert squares")
D: movement that preserves uniform velocity (e.g. moving diagonally, north/south, and east/west are all equally 'fast')
3e chose to take points A, B, and (essentially) D, and thus dropped C; you have to perform a calculation to account for the 'inefficiency' of moving diagonally. However, this calculation is imperfect; it makes every diagonal square count as 1.5 units, when it should be sqrt(2) = ~1.414, thus it technically violates point 4 as well. This effect is essentially imperceptible on the scale of typical battles, but if you assume that this is true, then there is a difference in efficiency between travelling diagonally and travelling partially by straight line and partially diagonally.
Each of the other requirements follows from the act of trying to discretize space. Real, physical space is not discrete (and no, quantum physics doesn't show it's discrete at the quantum level either, but that's an unrelated subject). We live with the Euclidean (L2) metric, where there is one unique shortest distance between two points, which requires solving a right-triangle to calculate and in all but a subset of cases (that is, Pythagorean triples) is an irrational number. Irrational numbers cannot be handled by a discrete, integer grid placed on space, so you have to "break" something.
Even hex grids aren't immune. They meet points A, B, and C perfectly, but they fail at point D. Depending on how you orient the hex grids, moving north/south will either be slower or faster than moving east/west--it depends on whether your hexagons have vertices on their north/south bisection, or their east/west bisection. Travelling NE or SE will never be less efficient than travelling due north/south or due east, but may be more efficient. Again, though, these considerations are small for the areas typically used by tabletop battle maps, which is why people like the hex grid, though it makes "square" structures a little awkward. A game like Civilization or Age of Wonders, where you may cover hundreds of hexes exploring or attacking, it adds up to enough to be worth caring about.
Because squares are simple to work with, and because people think in perpendicular lines better than they think in lines at 60-degree angles (by a small margin), 4e and games like it eschew being particularly "accurate" to real-space geometry. They value simplicity of representation over physical-world accuracy, assuming that it is an acceptable break from reality to make the game easier to run and adjudicate.
If you really want to blow your mind, you should check out the Manhattan distance (the L1 metric). In that, a circle of radius r centered at the origin is a "square," rotated 45 degrees (so that its "vertices" are on the axes), with 2r tiles on each "side." (I used quotes on "square," "vertices," and "side" because it's not a true square: it has jagged edges).