The Health and Human Services Secretary admitted he once shared the actor’s low opinion of the president