Malaysian police have detained more than 100 Rohingya refugees and asylum seekers who were seeking help from the UN.