Supporters of campaign finance laws have been apoplectic since the Supreme Court struck down a ban on corporate political ads in Citizens United. Having lost another big case this year in McCutcheon v. FEC, they now want to write their views directly into the Constitution.

Last week, the Senate Judiciary Committee approved a resolution calling for a constitutional amendment that would let government limit contributions to candidates and spending by and on behalf of them. The House will take up a similar proposal soon. To see where this amendment would lead if enacted, consider that the law in Citizens United prevented a group from distributing a film that criticized Hillary Clinton during her last presidential bid. During arguments in the case, the government’s lawyer admitted that the law could apply to books as well.

At the core of this effort is the very dangerous view that freedom of speech isn’t an inalienable individual right—a right to say what you want regardless of what others think—but a privilege that we exercise at the sufferance of “the public.”

This is the prevailing view among many intellectuals and politicians today. Justice Breyer expressed it in dissent in McCutcheon joined by his three colleagues on the left. In his view, the First Amendment protects the “public’s interest” in having its “collective speech matter.” Sen. Patrick Leahy, D-VT, echoed this view when he complained in advance of a hearing on the proposed amendment that the Court is ignoring “the voices of all Americans” in order to “amplify the voices of billionaires and corporations.”

This is a fashionable view—we need campaign finance laws to prevent rich and powerful “special interests” from drowning out the “the public.” But being fashionable doesn’t make it true.

After all, who is this “public” that allegedly isn’t being heard? Does it include newspapers like the New York Times and the Wall Street Journal? Broadcasters like MSNBC and Fox News? Millions of bloggers and Facebook and Twitter users? Donors to hundreds of political and advocacy groups? Jaw-boning politicians?

The fact is, “the public” doesn’t exist. Only individuals do. Terms like “the public” are never used to refer, literally, to every last person who lives in society. Instead, they are always used to justify one group of people using the force of law against others.

Consider how easy it is to define “the public” to mean groups whose ideas you favor. To Harry Reid and his allies, “the public” doesn’t include the Koch brothers or Tea Party groups targeted by the IRS, but it does include unions, environmental groups and others who support unlimited government power.

Of course, the right sometimes does the same thing. But that’s part of the point. The “public interest” can be used to justify any restriction on speech. No one ever admits that they want to silence others, so they invoke this magic concept to do that dirty work for them.

And make no mistake: silencing people is the point of campaign finance laws. Yale law professor Owen Fiss defended those laws in his 1996 book, The Irony of Free Speech, by saying that government may “have to silence the voices of some in order to hear the voices of others. Sometimes there is simply no other way.”

Silencing the loudest voices is the only way to make sure “the public” gets heard over individuals. And it’s always where campaign finance laws lead, because people easily find ways around them. Limit contributions to candidates, and people will spend money on their own ads. Prevent people from saying “vote for” or “vote against” and they will criticize candidates in other ways. That’s why, in 2002, Congress passed McCain-Feingold, which banned groups from even mentioning candidates near an election. No restrictions on spending for speech can be successful unless all spending is restricted.

The solution to this creeping censorship is to defend free speech as a right, not a privilege. That right does not protect our “voices” or guarantee that we’ll have “influence.” It protects our freedom to speak. Whether we are loud enough, persistent enough, or articulate enough is up to us.

Amending the Constitution is a long shot, and today’s effort will almost definitely fail. But tomorrow’s may not. If you value free speech, now would be a good time to start making yourself heard.