Holly Willoughby and her former co-star were once very close, seeing each other every week while working together, but their friendship appears to have grown more distant over the years