Right,rip it all off and tell me if you think its a bit noisy,only joking of course,mine is a 2 seat commercial so maybe they skimped a bit on building them.
I think it is probably a bit noisy even with it on but doesn't sound like the noise your describing. Only time I heard a chattering kind of noise like you seem to be describing is from an old fourstroke Honda XL 250 (still have it and its 25 years old) and that is almost definately a top end kind of noise, worn valves or bearing or something like that on the top of the engine.