A short essay on bad software reviews

By David Walker (Google profile)

How's this for a short review? The latest version of Adobe's award-winning Photoshop offers something for everyone from long-time print users to Web designers. With much-requested new features like multiple undo, editable text, "live" effects and better colour control, version 5.0 advances a tool that is already the industry standard.

The bad news is that I didn't write this review. Adobe did; I just edited Adobe's material a little. The words came from the 35 pages of material which Adobe sends to reviewers along with its Photoshop software - including a 20-page "reviewer's guide".

I don't want to single out Adobe, or even blame them. In a tough market, Adobe is just matching the tactics used by companies like Macromedia, Microsoft and probably dozens of others. Adobe's guide may not be objective, but it certainly describes the latest Photoshop well - at least up to a point. And you'd try to accentuate the positive too, if you were mailing $1500 worth of software to some bozo writer who might not know the difference between image-editing and word-processing.

Lazy reviewers are to blame

Now, reviewers don't have to read this stuff. We can put it all aside, install the software, play around with it and then decide for ourselves whether it's any good. But I don't know how many do. Sometimes a quick read will make you feel uneasy. A startling number of reviews seem to echo the material that comes along with those nice shrink-wrapped boxes.

For instance, many reviews document new features in loving detail. Yet new features don't matter all that much in many programs: most users don't use most of the features they've got in the old version, and features that are present are often hard to use. These days, new features usually matter far less than the ease of using software, or its suitability to the various tasks it's supposed to perform. And all of that stuff matters less than a tough value-judgement about which program suits what sort of user with what sort of bank balance. One prominent software writer, Daniel Will-Harris, has fleshed this issue out in a Web essay at http://www.will-harris.com/badrev.htm.

Software reviewing isn't easy

Perhaps it should be no surprise that software reviewers cut corners. As software gets more complex, really mastering a product takes longer and longer. A product like Microsoft's FrontPage, designed for several different types of users in different environments, and designed to interrelate with its own special set of "server extensions", takes serious effort to properly analyse. Comparing version 9 of the product with version 8, which you haven't used for 18 months, is even harder. And a program which takes weeks to learn can be easy to use once you've mastered it (many users would cite Photoshop as an example). Just to top it all off, a program like Microsoft's Outlook 98 can be worryingly slow on a Pentium 133 with, say 16Mb of RAM and suddenly silken with 32Mb - so how many machines should you test on?

Amid all these complications, publications often want reviews in front of the public as soon as possible. Software reviewers want to go home, see their friends and family and take a break from the 17-inch screen. And the industry standard often seems to be new features, some context, a couple of puns and a screenshot from the demo files on the program's CD-ROM. So why spend a week delving into the nooks and crannies of Internet Thingamajig 4.1, when a couple of hours will do?

What happens when software reviewers get sloppy? Sometimes users get stuck with software that just doesn't work as advertised - as did people who thought Microsoft's otherwise admirable Publisher 97 would produce decent Web pages. Sometimes users pay for more than they need - as, no doubt, have some people who read that Photoshop 3 was a required tool for making Web-ready images. Sometimes even the company gets fooled, as Sausage Software did when reviewers failed to pinpoint the yawning gap between its once cutting-edge HotDog Web editor and up-and-comer HomeSite.

Of course, I'd never write a bad review. And I'd tell you why - except I've already earned my cheque for this week ...