Yeah, prison planets and such are difficult because of all the evidence, plus the genetic similarity to other earth life.
That said, it's pretty easy to drop in an extra few thousand years of development in there somewhere. When you are looking at a species lifetime of 150K years or so, missing 4-6k years near the end isn't that tough. Also, since we don't know what finally sparked the BIG CHANGE (other than, perhaps, population density?) to where we as a species are now, it's reasonable to throw in that it actually happened a few thousand years earlier. Suspension of disbelief is easier, too, if it's not just "we forgot about these aliens" but actually "A huge and powerful galactic society actively destroyed our earlier society and worked to hide it from us".
At least I
think it's easier...