Cato has a study up by David Boaz and David Kirby that contains the most thorough analysis I've seen on libertarian voters, including estimates on how large of a voting block they are and how the block is potentially capable of swinging close elections. It is also pretty honest about some of the problems libertarians have in gaining political influence, especially their inability (or unwillingness) to organize. From the abstract:
Not all Americans can be classified as liberal or conservative. In particular, polls find that some 10 to 20 percent of voting-age Americans are libertarian, tending to agree with conservatives on economic issues and with liberals on personal freedom. The Gallup Governance Survey consistently finds about 20 percent of respondents giving libertarian answers to a two-question screen.