A former staffer and three other women accused Swalwell of repeatedly making unwanted sexual advances