The easiest thing for people who are making a website to do is clutter up your sidebars with ads and widgets. When you clutter up your sidebars with these things then you are making your website slow and clunky. There is nothing more aggravating on a visitors end than a slow website. You could have the best content in the world but if your site is slow it will push people away from it.
If you look at your site and think that you have too much stuff on it, then you probably do. Keep it simple. Not everyone cares about the time in your home or the weather where you live so don't put it on there. Text blogrolls are fine because it doesn't take long to load them. Imaged blogrolls take too much time. It is really a matter of common sense!
If you want to find out your website speed then take the website speed test.