Defenders of the possibility of time-travel usually address this potential contradiction by distinguishing between your personal time (the time kept by your biological clock) and external time (the time kept by the world's calendars). Your departure on a time-travel voyage can be future in your personal time (as well as in external time) even though your destination is past in external time (and future in your personal time).
This distinction is already required by Einstein’s special theory of relativity. If you travel in a rocket so fast that your personal time passes much more slowly than external time passes for residents of Earth, you may return after one year of your personal time to find that your generation has died off: think of it as time-travel into the future. Stories about time-travel into the past also require distinguishing between these two kinds of time.