California congressman and prominent Trump critic surprised ‘that it took him this long to come after me’