Tariffs on imports, mass firings of federal workers are some of the issues taking a toll on consumer psyche.