The remarks came after his interaction with Prime Minister Modi after an official programme in Chandigarh garnered attention on social media