This article is part of The D.C. Brief, TIME’s politics newsletter. Sign up here to get stories like this sent to your inbox.House Speaker Mike Johnson could be in trouble. Bigly.In just about every corner of his big-tent Republican Party, the affable Louisianan is facing angry members, many of whom view the House Speaker as part of the problem. As they brace for Obamacare costs to spike in a few weeks, and farm-state legislators find themselves trying to sell bailouts for soybean farmers struggling under President Donald Trump’s tariffs, Johnson is the GOP leader best positioned to help House members feel like any of what they are doing matters. But after less than a year as Trump’s Speaker, not many view him as up to the challenge. Even the most Trump-ian lawmakers are starting to have their doubts, as they complain that Johnson has stifled their voices and shrunk the chamber’s power in the process. And don’t even ask the women in Johnson’s caucus how they’re feeling unless you are ready for an epic—and not wrong—vent. [time-brightcove not-tgx=”true”]“Women will never be taken seriously until leadership decides to take us seriously, and I’m no longer holding my breath,” Rep. Nancy Mace wrote in a New York Times essay published Monday under the headline “What’s the Point of Congress?” Mace, who has opted to not run for another term and instead is working to become Governor of South Carolina, gave faint praise to Johnson as “better than his predecessor” but was unmistakably down on her job, and his ability to make it any better. As they enter into what’s looking like a rough midterm cycle, House Republicans find themselves struggling to explain what they’ve accomplished with their party in full control. A paltry 46 bills have been signed into law by Trump’s Sharpie, putting this Congress on track to deserve the title of the do-nothing-est Congress in a generation. (By contrast, President Joe Biden signed into law 274 bills in his final two years in office, and another 365 during his first two, according to GovTrack data.)If you’re Mike Johnson—or any candidate with an R after your name on the ballot next year, really—it gets harder to justify the ask for a vote given how little they have to show for their two years. Sure, they helped Trump pass his One Big Beautiful Bill spending package, but most of those provisions won’t be felt before Election Day and the more painful cuts to pay for them don’t kick in until after ballots are counted.Put in the plainest terms: Johnson is ruling the House on borrowed time. The modern GOP is the odd and unruly amalgam of competing interests, with the hawkish advocates of regime change breaking bread with the America First isolationists, the free-market absolutists giving space to the protectionist tariff flirts. Trump’s menace and protection, offered in equal measure, have kept most dissent at an arm’s length, but it’s getting shorter by the week. It took 21 days for Republicans to settle on Johnson in 2023, and even then only after some arm twisting and goading from Trump. If Trump changes his loyalty, the Johnson era could be over much faster—which may be fine with Johnson, who groused to a podcaster recently that the job was a big hassle, comparing it to being “in triage every day.” But Johnson may have a saving grace in that no other Republican is standing in the wings and few seem interested in the thankless job of serving Trump’s whims. Johnson is fast-approaching a tenure three times as long as the man he replaced, the deposed Speaker Kevin McCarthy who lasted all of 269 days in the job. That is little consolation for Johnson who has spent the first 10-plus months of Trump’s return to Washington getting battered from all sides.Johnson’s defenders note that he has proven himself an able fundraiser; last cycle alone he and his partner political arm raised $23 million to hold a narrow majority. He started this year by raising $32 million in the first quarter, and another $19 million in the second, and another $13 million in the third. Linked super PACs and advocacy groups, too, are hauling in the cash thanks to the prospect of unified Republican control of Congress and the White House. If one of the two main tasks of a Speaker is to bring in the bling and keep the natives content, he’s cleared at least one of those bars.Still, money doesn’t mask math. Of the 39 House members who have decided to call it quits, 23 are Republicans. As Puck notes, this puts the list of Congress quitters on track to beat the last one, when 21 Republicans were among the 45 House members who headed toward the exit. Privately, some are estimating another two dozen House members are ready to be done with Washington, especially given how Democrats have been overperforming all year in special elections and state contests. That’s not to say these GOP seats are all going to spin into the Democrats’ column. One lawmaker who is turning in her voting card a year early, Rep. Marjorie Taylor Greene of Georgia, is from a district that has a baked-in GOP advantage to the tune of 19 points. In a 60 Minutes interview that aired Sunday, Greene refused to call herself MAGA and cast her one-time pal Trump as a lost soul. And Rep. Don Bacon, a frustrated Republican and the only announced retirement from a swing district, says he will stick around to finish his term.Still, that abacus leaves Johnson—and Trump—with just a one-seat margin of error, at least until Georgia has a special election to replace Greene. (A date for that has not been set.)But even if there was the motivation for the House to do something meaningful, there’s also this reality that Mace distilled in her essay: “The obstacles to achieving almost anything are enough to make any member who came to Washington with noble intentions ask: Why am I even here?”It’s why many have decided not to be. And if they do stay, there’s no guarantee that Johnson will continue leading them. After all, as Rep. Elise Stefanik, a close Trump ally, told The Wall Street Journal last week, the growing view is that Johnson has proven to be unsuited for the job: “He certainly wouldn’t have the votes to be Speaker if there was a roll-call vote tomorrow.”Make sense of what matters in Washington. Sign up for the D.C. Brief newsletter.