If that's the character feel you're looking for, I'd suggest discussing with your GM a custom power that shifts physical defense to endurance. I'd phrase it as:
-1 Armored Hide: You may use endurance to defend against attacks or maneuvers that would inflict physical stress or damage - unless that damage would be inflicted in a way that bypasses your toughness catch. In addition, you may complement your defense with might when defending against large, brute force style physical attacks, such as an ogre's club, a boulder thrown by a giant, or a collapsing wall (but not against gunfire, the thrust of a rapier, etc.)
You'll still want a decent athletics, though. Catch-based attacks will still force you to dodge normally, as will many maneuvers; the power lets you use endurance against a "cut over your eyes" or a "bruised ankle", but won't help against non-damaging maneuvers like tripping, disarming, etc. It also won't help against grapples. And mobility is useful anyway.