For your working or recreational desktop environment, there's no definitive answer: it's all about how you use your system. More memory doesn't affect boot times in the right way, however having more things pre-cached should lead to a faster Windows experience once you sit down to work or play.
However task switching out of games, load times of save games, and even in-game itself - 4GB or more of memory makes the experience faster and smoother. We've got upcoming triple-A titles like FarCry 2, Stalker: Clear Sky or Crysis: Warhead, which will be at least no less intensive than what is currently available, so it's absolutely worth doubling up over 2GB if you're a keen member of the PC gaming master race. For a few less intensive titles like Call of Duty 4, it doesn't make a blind bit of difference between 2GB and 4GB of memory. Consider what you play before you dive into your next purchase.