Well true enough. Assuming there is a continuum of reality and unreality. There are no certainties in life.
The way it was looked at in ancient times is that reality was sort of like a cosmic egg, floating in an endless sea of chaos. If you consider it that way, the borders are the shell of the egg, and the sea is the Outside.
In a way, everything is a matter of definition. But for the purposes of the series, I think the Walls are very much a literal barrier with a fundamental difference to the matter of the Outside (for want of a better word...can unreality have matter? Mordite?)
The easiest and most clear definition would be to understand what existed before reality (if there even was a before...although Jim says the Angels predate Time itself - which makes the whole concept of before somewhat pointless). Cause and Effect have no place in such an existence. Which then would lead us to the nature of TWG and Creation itself, and it/their purpose. I would love Jim to write a fantastic theory that suits the rules that he has made, but I cannot be sure he will go that deep.
Language makes this whole thing more complicated. In many cultures and languages other than English, Past is not "before" and Future is not "after". Sometime Past is "left" and Future is "right", sometimes "below" and "above" respectively. And many others.
So as you imply, it comes down to how you look at it, in that way. But I also think the Author's intent should not be diminished or dismissed - and I think in a quantitative sense, he considers the Nevernever and our Universe distinctly and intrinsically separate. Food for thought!