True. I just reread it. Considering the prisoner said "The way you novices always do" strongly suggests he's had experiences with many Wardens. So been locked up a long time.
Yup, he has been there for possibly eons. Some things do point to him possibly being Merlin, though I think Jim is keeping his options open.
Many myths surrounding Merlin, one says he sleeps in a cave in Cornwall, England, some even say in crystal. Bulfinch's Mythology has an intriguing blurb about Merlin being trapped in a thorn bush by the Lady of the Lake and he sleeps there still,
"though his voice may sometimes be heard." So no thorn bush, but our prisoner seems to sleep a lot, demands to be left alone to sleep, and is sometimes heard. In fact he is the only prisoner that we know of that Harry has ever talked with. He is imprisoned in a cave and in crystal, so a lot of "hint, hint" that it could be Merlin, but they still may all add up to nothing.