24 April 2016

I got this question by email last Wednesday.

‘… Do we need more independent testers? Better proof of independence? Sites like this obviously aim to bring that evidence to the user from the user. But I see reviews for Endpoint solutions that I know are factually ineffective at catching threats ranking in the top 1-5 which is surprising…’

In my opinion, independent test results are a good starting point, but it makes no sense to rely completely on them, since the satisfaction with a security solution depends largely on criteria that cannot be simulated in test environment.

First of all, it is of crucial need to have a clear understanding of the threats and vulnerabilities one wants to mitigate with a solution. What are the risks the organization is exposed to? What are the threats? What threats do we want to address with a new solution? What risks do we want to mitigate with the solution?

Some hours of brainstorming in an interdisciplinary team are required to get this right. The resulting checklist is the basis for the further product pre-selection.

Once this is clear the pre-selection process can start. Most vendors claim to solve all security problems of the world with their solution, but in reality, most of the solutions mitigate few threats only. At least at the end of the pre-selection process it is very important to have an idea about the threats that a solution mitigates effectively.

With this, one has a good chance to find a solution that fits to his needs. The criteria used for pre-selection as well as the results can be easily communicated in peer reviews.

Unfortunately, there are some other factors which affect the effectiveness of security solutions and the ease of their implementation. Such factors include e.g.

• the existing IT landscape of an organization,
• the integration of the solution into the IT and security landscape of an organization,
• the integration of the solution into the business, IT and security processes,
• the maturity of the IT and security processes, and
• the skills of the IT staff.

The successful implementation and satisfaction with a solution depends in a large part on these factors. Therefore, these factors should be the basis for the final selection process. Unfortunately, these factors are mostly not communicated in peer reviews.

From my point of view more independent testers or better proof of independence will not solve the problem. It would be very helpful if we could simulate in advance how a security solution fits in an organizations existing IT and process landscape, and how a security solution affects the security level of an organization.

Ok, sounds like science fiction, but we all need to have some dreams, at least sometimes. Let us start with publishing the pre-selection checklists and information about the IT landscape, of course in anonymized form.

Have a good week.