Seth Nickell had an excellent post on the utility of usability testing.

Desktop software usually presents a substantially different problem space from web pages. Compared to each other, desktop software represents more complex and varied operations where long term usability is crucial, whereas web sites represent a simple operation (very similar to 100 other websites users have used) where "walk up and use perfectly" is crucial. Design of infrequently used software, like tax software, is much more similar to web site design. One simple example… In most web pages, learnability is paramount: if on the first time visiting a web site users don't get what they want almost instantly and without making mistakes they will just leave. Learnability is the single most important aspect of web page design, and usability tests (aka learnability tests) do a marvelous job at finding learning problems. In a file open dialog learnability is still important, but how convenient the dialog is to use after the 30th use is more important.