The material is an inescapable part of the city’s landscape and heritage — but its future is precarious