since the Time Portal normally gets to go back in your hand if you have the Time Traveler…

A: The Time Portal gets discarded. Since the Time Traveler doesn’t get to use the Time Portal at all, he can’t get it back again.

Also of note: The Time Portal causes your turn to end immediately, but if the Action gets canceled, then all portions of the Action get undone, so your turn also does not end early. It’s like the Surprise makes the Action disappear entirely.