I would apply the speed being halved only in the case of an unwilling target. Point is, I wouldn't use the grappling rules at all for a willing target.
Here is how I look at it in combat you would most likely be suprised and put up a fight at first because you may not realize who it is just my take from real life experience. Now if they are taslking smack and their friend dragging them away usally it is a haphazard fake attempt no movement penalty.
Just the way I view the siutations