Hexus has compared the GPU-accelerated video encoding functionality offered by ATI and NVIDIA, you can check it out over here. The author concludes that GPU-accelerated video encoding is significantly faster than using the CPU alone, but that the image quality varies hugely. Sometimes AMD-produced video is better, while other times, NVIDIA's looks better.
Looking for a cross-company performance comparison, finding applications that support GPU-accelerated encoding for both manufacturers has been rather elusive. Thankfully, a couple of companies, CyberLink and Roxio, have agreed to work with both AMD and NVIDIA's GPGPU technologies, and it’s now possible to conduct a fair comparison between the two. Starting off with the high-end and using CyberLink’s MediaShow Espresso application, costing £30, we’re pitting AMD’s recently released Radeon HD 5870 against NVIDIA’s closely priced GeForce GTX 295, setting them to transcode a 1080p MPEG-2 HD clip into a YouTube-compatible 720p MP4.