Testing cybersecurity products: A Q&A with AVLab’s Adrian Ścibor

When it comes to cybersecurity, understanding how well products do or do not work against malware is essential for both individual users and businesses. AVLab Cybersecurity Foundation, a leading entity in the field, dedicates itself to advanced ‘In-the-Wild’ malware testing, employing state-of-the-art methodologies and a global network of honeypots to evaluate the effectiveness of security solutions.

Adrian Ścibor is the founder of the AVLab Cybersecurity Foundation. As part of his cybersecurity activities, he is responsible for testing protection solutions against threats. In addition to testing work, Adrian also develops strategies and tools to protect data and systems from cyberattacks, and is a contributor to the Anti-Malware Testing Standards Organization (AMTSO), an international not-for-profit group which focuses on improving testing methodologies. In this Q&A with Adrian, we explore the nuanced world of antivirus and anti-malware testing, discussing how these evaluations are conducted, the changes they’ve undergone over the years, and their overall significance.

Emsisoft: In case anybody doesn’t know, what exactly is antivirus – or anti-malware – testing?

Adrian: In order not to go back to the testing methodologies of decades ago and go into detail, I will try to make it very simple and clear. In my view, it is about testing security technologies, whatever they are called today and however they work under the hood. These technologies have been integrated into IT solutions and systematically refined by vendors over decades. Testing the security features in different scenarios can reveal their weaknesses, for example when the antivirus engine is processing malware. Sometimes bugs are discovered that are not necessarily easy to spot in the real world, because users are not obliged to share detailed diagnostic data with the vendor, plus they do not generate as much diagnostic data at once to spot a needle in a haystack. Testing using algorithms for processing machine information catches these nuances. An important part of testing is talking to the vendor, the technical engineers, to point out a potential problem. Sometimes this problem needs to be discussed and, if necessary, a fix implemented, which will reach users in the next software update. Interestingly, sometimes flaws in the methodology are uncovered, which the vendor points out to the testing company. This is also an important part of improving testing processes.

Emsisoft: How has testing changed over the years?

Adrian: Testing is evolving, as are security technologies. We are aligned in our efforts against cybercriminals attempting to breach these security measures.
Testing has changed dramatically over the years, thanks to developments in technology, operating systems, applications and tools for testers. Today, most tasks (not all) can be automated, cloud computing can be used, more precise data can be extracted from telemetry data than before. There has also been a significant improvement in cybersecurity awareness, meaning more companies and users are demanding well-tested services and products from vendors. I can only speak for us – in a sense, the result of testers and vendors working together is a better quality of services and software, which then reaches the end customer and is installed in production environments or in our homes.

Emsisoft: Some people are concerned that tests must be biased because they’re usually paid for by vendors. How do you respond to that?

Adrian: I can understand these doubts, I used to also not know how testing is done from the inside and how much money it requires – from server maintenance, licence fees and associate salaries. In addition, we have already seen some serious incidents where a vendor has been caught manipulating results, which has only worked against the testing industry as a whole. Then there is the other side of the coin – so-called sponsored tests – these should be transparently labelled. In my opinion, this is being improved by press law regulations.

The fact that a vendor pays for a test is not a bad thing. It is the same as any other service, but can be regulated with additional technical, content-related requirements. The regulation of testing is dealt with by AMTSO (Anti-Malware Testing Standard Organisation), although you do not have to belong to this organisation to be able to test and give factual opinions. As I have already mentioned, an individual test should be labelled – it is paid for by the vendor in question, who expects marketing benefits in return, but it is quite another thing to cheat. I see no reason to agree that, in the event of a result that is not satisfactory to one of the parties, it will simply not be published. It stands to reason that the benefit will be derived anyway from pointing out bugs in the software and issuing an update. The same is true of articles and blog reviews, which are prepared for a fee.

Another situation is when the laboratory selects or invites vendors to participate in the test. In my experience, I can say that we do not charge for testing. We charge for our technical feedback, for the content development, marketing material used by the vendor (e.g. certificates, logos, other files). Vendors can choose not to participate in tests and can raise objections, so it is worth having everything documented in case of such discrepancies.

Emsisoft: If a product performs well in tests, does that mean it will perform well in the real world?

Adrian: It depends. That’s what different testing companies are for, they test software from different angles, so you don’t have to agree with one particular lab. That’s what different types of tests are for, to give a bigger, more comprehensive picture of what a solution can achieve in a production environment. If testers can bypass security measures, attackers might also find success in real-world scenarios.

What I am pointing out here is the nuance – the methodology. Often, after a test, bad information leaks to the press that product X or Y scored poorly. It’s important to review the methodology to determine if any security features were disabled during the test. I sometimes observe this and understand when someone wants to test a particular technology at the expense of another. Usually, I am against this kind of testing approach, because the vendor has not invested in technology development for decades to use it selectively.

Emsisoft: How should people interpret the test results? Does one bad score equal a bad product?

Adrian: As always, the devil is in the details. I will say that it depends on the test – as I mentioned before, you should pay attention to whether certain protection features are disabled in the test and how well does the product protect without them? I emphasize this again because you need to analyse the results against all the features of the protection software, as this is usually how it will work in a production system. In addition, the protection technologies of different vendors can vary significantly, potentially providing varying levels of information to administrators and users. So it is best to look at a product or service holistically or have specific minimum requirements prepared that the product must meet.

I encourage individual test analysis when choosing software. The effectiveness of protection is very important, and additional features are also important, such as quick help from the vendor, technical support, the program interface, speed of operation, supported systems, functions in the management console, e.g. alerting about an incident, automatic repair of these incidents, additional remedies, indication of vulnerabilities in systems, disclosure of poor user account configuration, integration with external solutions, e.g. SIEM, disk encryption, backup, etc.

Emsisoft: AVLab Cybersecurity Foundation is a member of the Anti-Malware Testing Standards Organization (AMTSO). What is that?

Adrian: In general, AMTSO brings together companies and experts who want to work together for the common good of cybersecurity. Testing organizations like our AVLab Cybersecurity Foundation are heavily involved in developing these regulations.

Even before joining AMTSO, I was initially skeptical about testing regulations until I experienced their benefits in practice. I like the transparency, if a testing company wants to achieve compliance with AMTSO guidelines. Thanks to AMTSO, our testing methods improved significantly, and interestingly, we were already meeting most technical requirements before we applied for membership. Now, through working with vendors, we have implemented changes and are systematically doing so, improving our tools and procedures. We now know better what vendors care about – not marketing fame, but that their product is constantly being improved for the benefit of end customers, regardless of the certificates they receive – because this is the form in which it is accepted to show the assessment to the public, which does not prevent us from going in another direction, although no one has yet come up with anything better. Through testing, we manage to pursue a strategy of better protecting customers, and we already have experience working with the smallest start-ups to the biggest players in the industry.

In every piece of software, sooner or later someone will find a way to bypass protection, find that technical nuance that reveals a weakness, a vulnerability. That is what testing and cooperation to fight cybercrime are for. Regulations are not always good, as normal life shows, but when you join an organisation you have to stick to the rules. It is good when they are good rules. As it happens, AMTSO has helped testers on many levels: from testing simple technologies to setting standards for enterprise-class products, and I am not just referring to recommendations for testing technologies that counter cybercrime. Thanks to AMTSO, we have learned lessons on how to better collaborate with vendors for the common good, this good is what the end customer gets.

Emsisoft: Most products score quite well in tests. How much do the minor differences in their results actually matter?

Adrian: Details are money. Let’s say product X dealt with threats 100% of the time and the other 99.99%. In theory, the two results are almost equal, right? However, one product allowed a situation where one too many incidents occurred, and in practice this can cost a company millions of dollars. This is why the differences are in the details.

Emsisoft: Have so-called ‘living off the land’ attacks made malware detection and testing less important?

Adrian: Whether something performs a malicious action on a system, even using legitimate processes and applications from the operating system vendor, we find out at the end of the incident. Considering how security technologies have evolved, I don’t see the need to argue about the difference between ‘traditional malware’ and ‘living off the land’ attacks (fileless, LOLBins), because malware can use legitimate processes, and with legitimate processes you can do the same thing on a system that malware does. Today, therefore, attackers and defenders alike have to make an effort. However, they have better tools at their disposal than they did a decade ago.

Today’s solutions must have more multi-layered protection than in the past, when these attacks were fewer, on a smaller scale. At the same time, solutions should be relatively easy to use, which is not at all easy for the developer and engineer department of the vendor. The solution should provide immediate identification of incidents from all endpoints, it must fix incident problems quickly and easily to support small and medium-sized organisations in the security process. Very importantly, the product today must not “fatigue with alerts” so that important incidents are not missed in time by security professionals in SOC (Security Operation Center) departments or front-line administrators in smaller companies.

Emsisoft: Some tests refer to pre- and post-execution detection, and detection times. What’s the difference and why does it matter?

Adrian: It depends on the solution, on the particular vendor’s approach to security – is it oriented more towards preventive protection, or is it still stuck in the traditional in detection? The difference between the split of pre- and post-launch threat detection is relevant in terms of technical nuances:

1. It may be of interest to people who are in charge of security, they pay attention to details.

2. For vendors who find out where the competition has an advantage, what to work on to better catch threats at a very early stage.

3. It’s not always possible to get great results blocking threats in the pre-execution phase, so this is another piece of information about whether and how well the product handles threats in the post-execution phase. In my opinion, this is extremely important, because it shows the real protection of the software in question against a threat that for some reason got into the system and was launched. I think this is where all the magic and precision of the vendor in detecting 0-day threats starts.

Emsisoft: Beyond tests results, what else should people look for when choosing a security solution? 

Adrian: Oh, this is a broad topic. We should clearly separate the selection of a solution for businesses, with a further division in the case of the size of the company in question and complex infrastructure, remote workers, etc. and between software for the individual user. In the case of the individual user, he or she may also be guided by the tests of the same business product vendor. But not always. I want to avoid complicating matters further by looking into details about modules not designed for non-commercial computers. Of course, common features still include the vendor’s support, reputation, test scores achieved, availability of the interface in a particular language, additional modules for key personnel e.g. banking protection, the price of the solution and support for different types of operating systems: Windows, macOS, Android, etc. Today, each of us may have more than one or two devices that we use, and often we also want to secure our entire household family.

When it comes to companies, I would look at the availability of functions in the administrator console for the specific requirements of the server-computer infrastructure. On integration with additional services, such as SIEM. On the support of protection with EDR for non-Windows workstations – macOS, Linux. For industrial systems, it is worth paying attention to the possibility of custom implementation of protection and covering network protocols with data flow monitoring. Added to this are remote management capabilities, isolation of computers with incidents, search for intrusion traces, automation of administrative tasks and so on. Of course, these are just examples. A detailed analysis of the need must be dictated by the prior development of requirements for a specific company, a risk analysis.

Wrap-Up

Navigating cybersecurity can be complex and requires more than just a surface-level understanding of products and their test results. As Adrian points out, the choice of a security solution should be informed by a comprehensive analysis, considering not only the test outcomes but also the specific needs, infrastructure, and operational environments of the end users.

AVLab Cybersecurity Foundation, through its meticulous testing processes and adherence to the highest standards, such as those set by AMTSO, plays a pivotal role in enhancing our collective defense against cyber threats. Their dedication to transparency, innovation, and cooperation with vendors ensures that the cybersecurity community remains a step ahead, providing reliable and effective protection for all stakeholders involved.