Product Evaluation: 10 things you need to know
when testing the bleeding edge of the information security.

This article is intended to fill the gaps often overlooked by people when architecting security infrastructures. The list below is squeezed out of our experience in testing technology products.


So, you decided to expand your security framework by adding a product to better control, monitor, or administer your IT environment. After looking at 2-3 behemoth companies and seeing that their prices do not quite correlate to the list of product features, you might want to see what else is available on the market and ... well, there are quite a few vendors offering dozens of products. You should easily be able to downselect to 3-5 products depending on the features you're interested in and then comes the evaluation. A common opinion that an evaluation just shows whether a products suits for a specific task does not fully apply to the information security area. So, here are some evaluation tips that will help you to better negotiate the deal and choose the right product.

1. Question every claim they make.

Most likely you will be discussing product features with two people, a sales engineer and a sales manager. The latter is usually the one who will make statements and the former will reply to your technical questions. If they also fly their CEO/CTO to talk to you, be alarmed that the three people may actually be their entire company (which raises concerns about unclear future). A list of product features should not be a focus of your meeting, instead target your discussion on how those features work and don't be satisfied with just "yes" answers. For example if you ask "Does your product encrypt stored data?" make sure they use strong encryption and the keys are properly managed. Note that the sales engineer will not be able to go deep into the woods, so you may want to talk to the actual product engineers. Vendors usually offer this option, if not, ask for it. The best time for this type of conversation is in the middle of the evaluation when you get familiar with their product. You should have a list of specific topics to discuss. Be as technical as possible.

2. Do not focus on just one product.

Research alternatives and let all vendors know that you are looking at their competitors. Do not mention any names though. If a vendor knows who their competitor is they will try to play the list of features amplifying every minor thing that their competitors do not offer. For this very reason, please read tip #1 again.

3. Take your time.

A week is not enough for evaluation. The vendor is interested to close the deal as soon as possible. They will usually help you setup their product in a standard environment, guide you through the evaluation and show the perfect results. Spend some time with the product without vendor's supervision and try all possible scenarios. For example, if they claim their product that can process X Gb/s of network traffic, try it. It may appear that it will start dropping packets at half that value if you throw multiple fragmented streams at it.

4. Talk to their customers.

The list of customer's logos that they show you on the third page of their presentation reflects the capabilities of their salespeople, not the brilliance of their product. Sometimes the list includes companies currently evaluating the product and the vendor says that it is either already in production or the deal is signed. Ideally, you would talk to a customer of theirs without them knowing it. If this option is not available, ask vendor for at least two references and make sure the customer they give you is not the one who owns 50% of their company.

5. Make sure the product is secure.

You don't want to introduce a whole new avenue into your environment. You are choosing a product that deals with security - make sure the product itself is secure. This is especially true for products that collect and analyze the data. Would you want someone to break into an appliance that is listening on a span port and has 6 months worth of private email data stored on it? So, perform a penetration test on the product, its user interface and communication protocols.

6. Show them every single fault.

It works miracles during price negotiation stage. Be aware though that by disclosing the bugs you find in their product you are essentially doing free QA for them. Of course, they will fix all of them in their next release. If you discover too many issues, the product is possibly not mature enough and by handing over the detailed description of all the bugs you could put yourself in the situation where you'll end up buying their product at a higher price later.

7. Be skeptical about "Hardware-based" or "ASIC-based" solutions.

The "hardware" may turn out to be just a 1U PC-based server (running off-the-shelf RedHat) that they sell you for triple the price of a comparable brand-name server. Ask whether you can leverage your hardware to install their software. As for "ASIC-based" solutions, they are not necessarily faster comparing to software-based. Custom ASICs usually come with limited internal memory while software-based solutions can utilize gigabytes of your RAM. So, for example an ASIC-based IDS sensor might only be able to store 128Kb of signatures .

8. Bits do not cost anything.

It does not matter for a vendor whether they support X or X+1 customers. If you ask for a discount, they will give it to you. On the other hand, beware that huge discounts may indicate that a company is in a poor financial state. All mature products in the same market space have comparable list prices. Beware of products with list prices in the range just below the average price - it might be their only advantage and those products may have significant functionality deficiencies.

9. Do not be sorry for a vendor.

There were projects when our evaluation results literally made people cry and beg to buy their products. One vendor even offered a 100K product for free, so they could add the company logo to the list of their customers. Remember, you are choosing the product to protect your assets and if it fails and expose your data - you are the one who will be in trouble.

10. Last, but not least: Be professional.

Vendors are people, too. Treat startup folks like you would a CIO of a Fortune-50 company. Leaving aside ethics, vendors can also play your behavior against you, for example, by complaining to your senior management that they were not treated well. It is a small world and there is a possibility that a vendor has tight connections to the CIO of another company who will then talk to your CIO about it. Always have a well argumented way to explain why the product is not suitable for your environment.

Discuss this article in our forums