Pretty much, yeah. I know it's ignored for pulp and fantasy (which is fine, I love that stuff) but there's a reason weight classes in combat sports exist. Size and strength are /important/.
In real life, the small, fast guy doesn't dance around the big guy and kick his ass. At least not unless he -vastly- outweighs the big guy in skill. He gets his ass kicked. Not fair, but how it is. Not to mention that if someone is trained as a fighter, they will have the right kind of musculature for speed as well. Speed is, after all, based on your muscles (I'm simplifying a lot).