It's 50/50; I never ran any antivirus software on and off for year's, I still dont think I run any on my Media Centre, yet I have never had any problems with Virus's.
Although having said that, I have had Virus's in the past, just nothing fatal.
The one thing is though, I have seen the effects of fatal virus's many time in the past, from messed up computers to keyloggers taking someones bank details, to the likes of the CIH virus, which has rendered the firmware in some devices useless.
Its quite true that if you don't visit dodgy sites, you will usually be okay, but virus's can be spread in various ways, and with too much ease.
As for the likes of Norton slowing down you're PC, all the big boys do because they are coded shite imho; go for better, more accurate and more secure antivirus like Nod32, which whilst offering the very best protection of any software and being used by IT professionals worldwide, it very very light on system resources, to the point that you don't even know its running.