Away from The Great British Sewing Bee, judge Patrick Grant previously dated a famous face whom he ‘used to argue with a lot’