So your suggestion is that Outsiders are merely aliens from others planets in other alternate universes, who use some sort of connection between their Never Never and ours to reach our Earth because although their planet and ours don't share the same geographic location, they share the same symbolic "Center of the Universe" location?
Unless Outsiders come from an alternate universe made of anti-matter and/or a galaxy of said alternate universe made of the same (since current science can't tell which of the galaxies out there are matter vs. antimatter), it certainly wouldn't explain the properties of Mordite. Also, it doesn't really explain why they are so hard to affect with magic, if they are so symbolically linked to us.