Logging and alerting. Every computer security product should do lots of logging. What format are the logs in? Can you manipulate the log format, data collected, and export to other formats? Can the logs be saved to an external database, emailed to people, and restarted or recycled on a scheduled basis? How many different ways can alerts be sent (email, SMS, pager, network message)? Does it "message throttle" alerts so that dozens or more aren't sent for a single related event? Does the product allow you create custom alerts or to ignore alerts you don't care about?
Reporting. Most products shine or show their weakness in their reporting. You want a product with lots of built-in, canned reports. The product should allow customization of current reports and easy creation of new ones. Can reports be extracted; if so, to what formats? Can reports be scheduled on a periodic basis and automatically emailed to interested stakeholders?
Performance. Performance can be the hardest measurement to take in a test environment, since you won't have the same scale as the production environment. But don't simply rely on the vendor's claims. Interview other vendor clients of similar size to hear what they have to say about the product and its performance.
Make sure you get in writing what hardware specifications will get you peak performance. Many buyers put a clause in their vendor contract that requires a certain minimum guaranteed level of performance, both on server and client side, or they get their money back.
Support. Most vendors claim to have 24/7 support. But does calling the support number result in having to navigate an endless phone tree? Are the support engineers well trained? How many support calls are you allowed each year? What does it take to get escalated to advanced tiers of support? What would it take to get the vendor back onsite to resolve an operational issue? Again, don't rely on vendor claims. Talk to other customers about the same size as your company. Bigger customers often spend more money with the vendor, which often results in "special" care that a smaller customer may not receive.
6. Get references — and call them
Ask the vendor for a list of customer references, hopefully of the same size and in the same industry as your company. Although customers are nearly always selected for their undying love of the vendor, I find you can coax actual experiences from them.
My favorite trick is to wait until the end of the conversation, after they've told you nothing but great things about the product. Then close by asking, "What do you wish the product did or did better?" Often, customers will proceed to share their real concerns. I've learned about many deal-breaking problems this way.
Sign up for Computerworld eNewsletters.