We know that Google has said that plus ones can have a positive impact on your rankings but we also know site speed can have a negative impact on rankings.

A WebmasterWorld thread has reports of webmasters pulling the Google Plus One button from their sites because they have been one more element to slow down the page load time of their pages.

One webmaster said:

Truth be told, there are ways to speed up the buttons through some tricks. Michael Butler at RustyBrick (my company) posted an article named JavaScript Hover Effects to Speed Up Page Load Time. Basically, you do not load the buttons until a user places their mouse over the button. Before the user does that, the button is simply an image.

One webmaster claimed that his rankings slipped after adding the plus one button and then shortly after removing it, the rankings came back. Of course, there can be other reasons and factors, but this is what he said:

The last time we covered a topic like this was with the Facebook like button where a Googler said it will have no impact on your rankings.

Forum discussion at WebmasterWorld.

Update: Hours after I posted this, Google announced a faster Google Plus One button. Clearly this post had nothing to do with them releasing it, I am sure it was in development for a while.