... Arthurian you are talkin 4th Century AD the Castle is likely to be not that old.
Historically speaking, that "big stone castle" doesn't exist in England at the time of Arthur & Merlin (all our pop-culture imagery is just wrong on this). The stone castle is an import from the Norman conquest; best records say the "White Tower" (at the Tower of London) was the very 1st stone castle in Britain.
Romans built some of their
forts with stone; but that's a different thing, and Harry's
castle is decidedly NOT one of those!
That said, LOTS of Anglo-Saxon castles were "upgraded"
in situ to stone from earlier wooden structures, so there's no reason a Merlin-era castle -- with all sorts of magical import -- couldn't have been re-built in stone after the Norman invasion (and as we know, Merlin laughed at White-Council style "Law of Time." He could have come in with his enchantments far after the "Arthurian" era; in fact, that's clearly what he did(*)).
And since we don't know the precise origin of the castle, it may not even have come from Britain... or even Europe!
(*) Unless, of course, he
didn't! For example... what if it was built in the pre-Norman era by Merlin who simply led a team of masons & stonecutters & laborers & such from the High Medieval via Ways Through Time, to build it in some out-of-the-way location; then enchanted it (lots of runework, after all; lots of enchantments!) for an aura of "spooky & uncomfortable" and "tend to forget about the castle" &c. The Citadel of Aleppo was a stone fortress contemporaneous with Merlin. I'm unclear how-early Rashid was, or whether Rashid ever overlapped with one of Merlin's time-hopping excursions; but (hypothetcially) Rashid could have shown Merlin the middle-Eastern stone castles, and Merlin would have been all "I have
got to get me one o' these!"