ExtremeTech explains the basics of overclocking your entire system in their new Overclocking Guide 2010, you can read it over here.
Today, though, careful overclocking is not only reasonably safe, many enthusiast hardware manufacturers actually expect you to do it. Where manufacturers used to use bold letters to let you know your warranty was null-and-void if you overclocked a chip, nowadays many offer overclocking utilities right on their Web sites, or even in their product boxes. Intel motherboards have traditionally been the most conservative when it comes to overclocking, but today's Intel enthusiast motherboards have plenty of settings for juicing up your system, and the Turbo Boost feature found in high-end Intel CPUs actually automatically overclocks individual cores when other cores are idle. AMD offers a software overclocking utility for its CPUs called AMD Overdrive, and utilities to overclock Nvidia and ATI video cards are included either right in the driver, or available for download from the manufacturers' sites.