The actor broke down the "terrible" experience, which took place his first day filming the series, on "Watch What Happens Live" Monday.