An excellent WebmasterWorld thread has one webmaster who noticed he was accidentally linking to thousands of 404ed internal pages. He believes due to that issue, his site’s rankings have degraded in Google.

To me, it doesn’t sound far-fetched. I wouldn’t call this a penalty, but rather a major internal linking issue. Internal linking is important for many reasons, two of which include giving search engines a way to discover your content and showing them what your content is about using anchor text. If you link to pages that don’t exist, well – that cannot help you or the search engine. Forget the importance of using internal linking for other reasons, discovery is the basic purpose.

PageOnResults in the forum also added his personal experience with such an issue:

It happens, and you can recover.

Forum discussion at WebmasterWorld.