The downside to having a build server retain build history and every output it's ever built is the cost of disk space to store all of that output. Some people might just clear down their outputs directory regularly, but for those requiring a more controlled appropach, try the following service;
http://notgartner.wordpress.com/2006/05/22/release-build-clean-up-service-for-team-foundation-server/
[Edit: Of course now that TFS 2008 is released, this isn't an issue - it cleans up build history based on settings]