Justice Stephen Breyer needs to get with the program. At least that’s what many on the left think after last week. In remarks made for Harvard Law School, the Supreme Court justice threw cold water on progressives’ desire to expand (i.e. pack) the court. His remarks gave several reasons against it, saying he wished to get those favoring such measures to “think long and hard before embodying those changes in law.” Adding to his transgression was timing; these critiques came out the same week that President Joe Biden took a small step toward court expansion, establishing a commission to study it among other proposals for judicial reform.

Many progressives set a high priority on adding justices to the nation’s highest bench. Packing the court became popular on the political left in reaction to President Trump’s three SCOTUS appointments. It grew alongside progressives’ frustration, outrage, and fear that attended each one: Frustration at what they considered a stolen seat, held open by Mitch McConnell and given to Neil Gorsuch; outrage about Brett Kavanaugh’s alleged (but unproven) deeds as a teenager; fear as the additions of Gorsuch, Kavanaugh, and Amy Coney Barrett seemed to shift the court’s decades-long balance against the Left.

We should not lightly dismiss Breyer’s opposition. Breyer was nominated and appointed by President Clinton. He generally votes with other Democratic nominees, especially on matters of abortion, religious liberty, sexuality, and regulatory power. The Left believes these precedents hang by a thread, ready to be cut by the Court’s six GOP-appointed justices unless reinforcements are added. One might assume, then, that Breyer would welcome the help.

Breyer does not dissent because he disagrees with the progressive doctrines. To understand the divergence, we must consider where his viewpoint differs from his ideological compatriots.

Our political discourse often sees arguments as legitimate or not based on the viewpoint of the speaker. Does the person possess the “lived experience” earning her the right to discourse on a subject? A version of this position exists on the left and the right. One cannot speak about the death of George Floyd unless you share his race, those on the left say. And on the other side, some insist that none outside of the “deplorables” can assess the merits or demerits of Trumpism.

This inclination, however acted upon, stems from a legitimate insight. People often know better what they have experienced rather than what they have distantly observed. They know better because they have done more than simply live within a certain environment. That environment has contributed to their formation and has helped to create their “lens,” which then influences their thoughts and deeds.

The Constitution recognizes and makes use of this truth. In Federalist 51, James Madison noted the problem ambition plays in any government not administered by angels. Separating out governmental functions into discrete branches was a good start. Madison saw in the Constitution’s set-up that “[a]mbition must be made to counteract ambition.” In theory, separated institutions meant no one person or group would exercise all political power; all would compete with the others from within their own branches of government.

Merely commanding such separation, however, did not go far enough. Persons across institutions might share viewpoints that encouraged their ambitions to cooperate across institutional lines. Rich members of the executive branch, for example, might entice the same kind of persons in Congress to unite in purpose. These connections might consolidate power between branches, undermining in practice what was sought in theory.

Madison continued that good government needed an additional element, one wherein “[t]he interest of the man must be connected with the constitutional rights of the place.” In other words, officeholders must see their own ambitions tied to protecting and enhancing their institution in distinction from other perspectives or entities.

Creating this institutional loyalty required cultivating an institutional perspective. Congress, the presidency, and the courts do not just exercise different functions. Their doing so entails distinct perspectives and divergent tendencies. Congress, the Founders thought, would be defined by traits tied to deliberation that better equipped them for lawmaking. The unitary executive, to best enforce law, would tend to be a man of decisive action, what Federalist 70 called “energy.” And the judiciary would be defined by “judgment,” meaning the application of law to resolve disputes between litigants.

Returning to Justice Breyer, we must see that he argued not as a progressive but as a judge. He showed his own institutional formation by calling to preserve, against court-packing, certain traits that attended judgment’s exercise. At their core, these traits emphasized the judiciary as the branch that exercises reason in a distinct fashion. We can see this point in Breyer’s concern that courts should not be seen as “political.” It is fine for Congress or the president to act in a partisan manner, as the Constitution envisions them acting in part based on their will as an expression of the people’s will. Judges, by contrast, exercise their own reason to apply the will of another — the legislature and ultimately the people — not their own. Judges, tasked with applying laws passed by others, should seek at least the appearance of being disinterested in policy outcomes.

Moreover, Breyer drew on Federalist 78 to make the case that the court relies more than other branches on how its use of reason is perceived by others. Lacking power over money, like Congress, or violent force, like the executive, the court must persuade other institutions to follow it by the seeming legitimacy of its reasoning about the law, especially the Constitution.

Breyer claimed that adding justices in reactionary partisanship would undermine the perception of the court so vital to its effectiveness. In so doing, he argued against actions that would aid progressive causes in favor of ones that would preserve the court.

None of this is to say that Breyer is the beau ideal of a judge. Contrary to his Harvard remarks, his reasoning and voting on the Supreme Court often seem to sneak in exercises of will, and progressive ones at that, in applying the law. Still, this concession might show the wisdom of the Founders even more. For the office of judge, while not completely forming him, seems to have restrained Justice Breyer in a way it would not have Senator or President Breyer.

Thus, the Left is wrong here about Justice Breyer. On court-packing, he doesn’t need to get with the program. They do.

Adam Carrington is Assistant Professor of Politics at Hillsdale College and Garwood Visiting Fellow in the James Madison Program at Princeton University.