Donald Trump told reporters that Donald Trump Jr. wanted his father to attend, but that it was going to be a “small private affair”