Speculation has been mounting throughout Harry’s week-long UK trip that Meghan and their children would join him