You don't have to go after the person killing you - and he doesn't have to be right there...
For example, the White King wasn't there when he got hit - but then again that sounds like a normal spell that was being done, only to be "rushed" when Harry's mother added the shifts from her Death Curse to finish it.
Richard