For example, next week I'm doing a demo of a set of products that in the production environment will require seven different servers. Do you want to mimic all seven servers in the test environment, or is it acceptable to merge roles? If you can, you always want to mimic the configuration that will go into the production environment, but you can sometimes get away with fewer machines (or VMs) without sacrificing accuracy.
Think about naming conventions in the test environment. I usually recommend that the test environment names should be at least slightly different than those of the production environment. Why? Because many times the supposedly "isolated" test environment ends up having one or more connections into the production environment; if you use the same names, you could cause real operational issues.
Write down how you configured your test environment with enough detail so that anyone relying on your results could re-create it. If you're using virtual machines, now would be a good time to take a VM snapshot. That way if you're testing multiple products for the same solution, you can make sure you start with a clean slate for each test. I also usually take another snapshot just after the computer security software is installed, so you can go back to the original install state if you decide to change configuration options and test again. Create at least one test image for each platform the product must support.
5. Structure and perform rigorous tests
Once you get a test environment locked down and the product in hand, it's time to begin the evaluation. You can get some pointers by reviewing past InfoWorld Test Center reviews. Our reviews tend to break up product evaluation into a handful of categories:
Installation. You'll be evaluating installation on both the client and server side. How many different install methods are supported? Can you push out installs within the product itself or do you have to accomplish that using another method? What services must be running and what firewall ports must be open in order to install the software remotely? If you test installs, how many failed? Be sure to install at least once for each desired platform and form factor.
Configuration. Note how hard the product was to configure during or after the install. The best products walk you through a series of wizards that help you make the best choices. How hard is it to change settings afterward? How long before the changes take effect? Does the product offer both agent and agentless installations?
Management. How do you connect to the management console? Hopefully it's over a HTTPS or other secured connection. Does the management console allow you to create different access control views for different types of administrators? What remote clients does it support?
Sign up for Computerworld eNewsletters.