(Lincoln Agnew/for The Washington Post)

E.J. Dionne Jr. is the author of “Why the Right Went Wrong: Conservatism From Goldwater to the Tea Party and Beyond,” from which this essay is adapted. He is a Washington Post columnist, a Brookings Institution senior fellow and a professor at Georgetown University.

The Republican campaign for the 2016 presidential nomination has become a carnival, a comedy hour and, for many in the party, a horror show. Donald Trump and Ted Cruz, the two leading candidates in Monday’s Iowa caucuses, are despised by most of the party’s leaders. The GOP’s so-called establishment has, so far at least, been hapless in bringing them down. The closing days of the campaign here have been dominated by high-school-level hijinks around Trump’s refusal to participate in a debate because he didn’t like one of the moderators. Oh yes, and the debate was sponsored by Fox News, the GOP’s sacred cable network.

Democrats can be forgiven for being gleeful. At last, it seems, the GOP’s extremism and obstructionism throughout the Obama years are catching up with it.

But at the risk of spoiling the fun, the crisis of conservatism is actually a problem for all of us — and I say that as an unabashed liberal. An intellectually vibrant conservatism is essential to a healthy democracy. The United States needs conservatives willing to criticize the grand plans we liberals sometimes offer, to remind us that traditional institutions should not be overturned lightly and to challenge those who believe that politics can remold human nature.

Republican presidential candidates weighed in on immigration, the Islamic State, criminal justice reform and - of course - Donald Trump at the Fox News debate in Des Moines, Iowa on Jan. 28. (Sarah Parnass/The Washington Post)

At its best, as Philip Wallach and Justus Myers argued in National Affairs , conservatism is a “disposition” that “has the most to offer societies that have much worth conserving.” Even those of us who are critical of our nation’s injustices and inequalities can agree that the United States is such a society. The task of conservatives, Wallach and Myers write, is to offer “incremental adaptation” as an alternative to radical change.

The trouble is that large parts of the American right are not interested in “incremental adaptation,” and they certainly don’t want to compromise with Democrats. In a two-party system that frequently divides a government with separated powers, this produces exactly the sort of dysfunction that voters are so angry about. In the meantime, working-class Republicans are increasingly (and justifiably) indignant that their loyalty in election after election has brought them no material benefits and few satisfactions of any kind.

The rise of Trump is not an accident. Erick Erickson of the popular RedState blog was succinct: “The Republican Party created Donald Trump, because they made a lot of promises to their base and never kept them.”

It’s a problem that goes a long way back — to conservatism’s embrace of Barry Goldwater. Not the avuncular Goldwater who developed real doubts about the far right late in life, but the firebrand who built his 1964 candidacy on the conviction that the Republican Party’s establishment had betrayed its conservative loyalists. The Goldwater who taught his followers that “extremism in the defense of liberty is no vice” and that “moderation in the pursuit of justice is no virtue.”

For half a century, the history of American conservatism has been a story of disappointment and betrayal. Conservative leaders have denounced decades of change, pledging what would amount to a return to the government and economy of the 1890s, the cultural norms of the 1950s and, in more recent times, the ethnic makeup of the country in the 1940s. But no conservative administration — not Richard Nixon’s, not Ronald Reagan’s and neither of the Bush presidencies — could live up to the rhetoric that conservative politicians regularly deploy to rally their supporters.

Conservatives in power could never materially reduce the size of government, because so much of what it does and spends money on — from supporting the elderly to protecting consumers to providing for the common defense — is so popular. Conservatives haven’t been able to roll back cultural changes, because most Americans don’t want to return where we were before the rights revolutions on behalf of African Americans, women and gays. And politicians can’t reverse the fact that white Americans gradually are losing their majority status in an increasingly diverse nation. The absurdity of Trump’s promise to deport 11 million undocumented immigrants is obvious to most Americans, if not to Trump’s supporters.

Conservatives have still managed to win elections, of course. It helps that they can count on older Republican voters to turn out, even in non-presidential years. That aging base, however, bodes ill for the party’s long-term prospects.

And the victories Republicans have won over the decades have produced neither the lasting electoral realignment that conservative prophets keep predicting nor the broad policy changes that the faithful hope for. For the rank-and-file right, the sense that their leaders have failed them and the political system shortchanged them has created a cycle of radicalization.

With each disappointment, movement conservatives have blamed moderation and advanced an ever-purer ideology, certain that doing so will eventually bring them the triumphs that have eluded them over and over. Nixon infuriated conservatives who believed he harvested their votes only to pursue liberal policies on a host of issues — from regulatory and environmental legislation to a realist foreign policy that took him to communist China. The collapse of Nixon’s presidency led to the rise of Reagan. Reagan, in turn, left intact many of the legacies of the New Deal and the most important elements of Lyndon Johnson’s Great Society. Conservatives rarely confess to disappointment over this. Instead, they hold these apostasies against his successor, and the defeat of George H.W. Bush led to Newt Gingrich’s revolution.

The partial exception to this pattern is George W. Bush, who briefly preached “compassionate conservatism.” But the idea never took hold in his party. It became at best a sideshow after the 9/11 attacks. And when the country turned on the Bush administration, conservatives said his problem was that he had not been conservative enough on spending, immigration, education and Medicare. A return to the true faith was the only prescription on order. The result was the tea party, which was a reaction as much to Bush as to the election of Barack Obama.

In trying to explain the rise of Trump, many conservatives insist that the self-involved billionaire’s unforgivable sin is his failure to be true to the conservative creed. “The case for constitutional limited government is the case against Donald Trump,” Ben Domenech argued in National Review, while the editors of that magazine declared, “Trump is a philosophically unmoored political opportunist who would trash the broad conservative ideological consensus within the GOP in favor of a free-floating populism with strong-man overtones.” My colleague George Will has suggested that a Trump nomination would “mean the loss of what Taft and then Goldwater made possible — a conservative party as a constant presence in U.S. politics.”

They’re right, of course, that Trump’s overriding ideology is opportunism. But in a perverse way, his rise embodies the phenomenal success of Goldwater’s war on moderation. If the GOP front-runners are uncompromising, this is exactly what most of those who remain in the Republican primary electorate want them to be. And this distinguishes Republicans from Democrats. A typical and important poll finding: In 2013, the Pew Research Center asked Americans whether they preferred elected officials who “make compromises with people they disagree with” or those who “stick to their positions.” Among Democrats, 59 percent preferred compromise-seekers; among Republicans, only 36 percent did. It’s no wonder that to chase the voters who have repaired to Trump and Cruz, Marco Rubio has been willing to blur his image as an optimist by ratcheting up the anger in his rhetoric.

The spectacle of what is happening in this year’s nominating contest already alarms many Republicans, but it will probably take a third consecutive presidential election defeat to force a real reckoning. Political movements, after all, tend not to change course until they have no alternative. The British Conservative Party turned to moderation and modernization under Prime Minister David Cameron only after three losses. The rule of three may apply here, too. And if the GOP lost again after picking a nominee who represented an unvarnished version of the conservative creed — Cruz comes to mind — it would no longer be possible to blame defeat on a lack of ideological purity, a charge conservatives leveled against both John McCain and Mitt Romney.

Would a Cameron-like course correction amount to selling out conservative principles? Well, yes, if conservatism is defined as resistance to all change — or at least to any changes that were once or are now being proposed by those seen as liberal or progressive.

But this is not conservatism. It is reaction rooted in deep pessimism that is entirely out of keeping with the American character.

With their “take our country back” rhetoric, conservatives these days do not seem to like the raucously and creatively diverse United States that actually exists. They refuse to acknowledge that certain reforms were adopted and then broadly accepted precisely because they better reflected the purposes of the American creed — “life, liberty and the pursuit of happiness” and the idea that all are created equal — than did the status quo.

They should realize that their forebears eventually embraced what reformers achieved. Many conservatives in the pre-Civil War period opposed the abolition of slavery; many conservatives in the 1930s opposed Social Security; many conservatives in the 1960s opposed civil rights laws. But the justice of these measures became obvious over time, and the values behind them became part of the American way of life. In this moment, conservatives need to ponder whether 10, 20 or 50 years from now, Americans — including conservatives — will feel the same way about same-sex marriage or the guaranteed, universal availability of health insurance.

Conservatives certainly do not have to “acquiesce” to every liberal reform, wrote the political theorist Greg Weiner, an admirer of the first conservative, Edmund Burke. But in examining the order that progressives have brought into being over the past three-quarters of a century, he argued that they do need to “approach its modification with a degree of caution and regard.” Undoing the entire New Deal/Great Society legacy is not, in the end, conservative.

The rise of Trump has called forth some useful soul-searching on the right. Conservatives need to look candidly at the roles of racial reaction and white backlash in building their movement and their continuing roles today, as witnessed in Trump’s xenophobia. They should recognize that today’s dynamic, tolerant and diverse America arose out of the very concepts of liberty and opportunity that conservatives have always extolled.

And they should acknowledge that a conservatism focused so obsessively on lower taxes for the wealthy and deregulation of the economy has little to say to the Americans with modest incomes who form Trump’s base. National Review, the founding magazine of Goldwater conservatism, acknowledged this month in its widely debated “Against Trump” editorial that if conservatives “cannot advance a compelling working-class agenda, the legitimate anxieties and discontents of blue-collar voters will be exploited by demagogues.” That’s true. But if conservatives continue to put relief for the “makers” and “job creators” at the heart of their governing program — as all their presidential candidates are doing this year — such voters will continue to wonder how seriously the right’s politicians are taking their struggles, their fears and their aspirations.

The country and not just the Republican Party would be better off if this very strange election year marked the beginning of a large-scale reassessment by conservatives of the trajectory their movement has been on since Goldwater transformed it in 1964. It is common for conservatives to say that liberals need to free themselves from the 1960s. This is now imperative for the American right.

Twitter: @EJDionne

Read more from Outlook:

Carlos Lozada: Could you trade your political beliefs for their exact opposite?

Why conservatives start culture wars and liberals win them

Memo to Sanders and Clinton: Even insurgents need an inside game

Five myths Donald Trump tells about Donald Trump

Follow us on Facebook and Twitter.

E.J. Dionne writes about politics in a twice-weekly column and on the PostPartisan blog. He is also a senior fellow in Governance Studies at the Brookings Institution, a government professor at Georgetown University and a frequent commentator on politics for National Public Radio, ABC’s “This Week” and NBC’s “Meet the Press.”