He says the Speaker’s constitutional position required him to remain above caste and religious considerations