Harry describes himself as a "thug" early on.
Presumably the stay-at-home wizards are much better at all kinds of wards, enchantments, illusions, veils, shape changing, astral projection, telekinesis, pyrokinesis, thaumaturgy, shapechanging, whatever.
although gold is a completely differently element than lead, precious stones like diamonds and emeralds are just common elements arranged a certain way. maybe they can do that with magic. transmutation.