Okay, how about this.
Each time you travel back in time, assign Time a stress track. The farther back you go, the fewer stress boxes. This stress track doesn't clear until you return to the present or die.
At the end of each scene, roll a weapon 0 attack against Time. The more you've done to affect the past during that scene, the more accurate the attack. Time always rolls 0 to defend, but players can tag or invoke Aspects to boost its roll or to reduce their own.
If Time would be taken out, it instead takes consequences. It has an infinite supply of these, but nonetheless uses as few as possible against any attack.
A mild Time consequence is a small aesthetic change to the present. For example, a child is named after a time traveller or a song that would have been written instead isn't.
A moderate Time consequence is a meaningful change to the present or a little bit of damage to reality. For example, a local election goes differently or some children with no ancestors come into existence.
A severe Time consequence is a noticeable change to history or meaningful damage to reality. For example, Gore wins the election against Bush or the laws of physics are altered in a certain forest.
An extreme Time consequence is an alternate timeline or major damage to reality. For example, the Nazis win WWII or a Walker enters the world.