Physicists, mathematicians and philosophers have been exploring the possibility of going forward to the future or back to the past