Can you drive a car safely at full speed while also doing activities that you could do while walking a normal speed? Seems like distracted driving at best to me. Flying a broom at normal speed doesn't require as much focus and concentration as driving a car down the freeway IMHO. In any case, a replacement of speed is a modifier to your speed, you cannot move using the broom land and then move your normal speed.But RAW it doesn't replace your speed. It moves, and you ride it. It's a vehicle, like a ship.
So I disagree. Feel free to rule differently.