Thankfully, there are generally enough trappings of a skill that it would take more refresh than it's really worth to replace a skill completely with another. It's almost possible with the combat skills, but paying points so that you can say "I hit him" with a different skill doesn't change things that much.
You could, for example take fists 5, and then take the following stunts:
Dodge with fists (replace dodge trapping of acrobatics)
Armed Arts (you can use fists to attack with 3 different types of weapons)
But you lose out on the abilities from athletics: climbing, running, jumping, and breaking falls
And from weapons you lose the ability to throw things (like knives, spears, and hand grenades)