Tom Hiddleston and Taylor Swift briefly dated back in 2016 but the pair called it quits after around three months together and he's attempted to shy away from discussing her publicly