Although Prince Harry admitted speaking out "annoys some people", the Duke of Sussex stressed his "conscience is clear" following the meeting with his dad, King Charles