Mitre Round 4 involved detailed testing of endpoint security (XDR and EDR) protection technologies against simulated attack scenarios based on the Wizard Spider and Sandworm threat groups. These evaluations covered both detection and protection (endpoint prevention) capabilities for each participating vendor.
For MITRE Round 4, twenty-one security solution vendors were evaluated in the protection test, and of those one vendor failed to fully complete the testing sequence (due to blocking an action that MITRE deemed should not be blockable). In total seven vendors claimed a 100% success rate, in which 9 of 9 tests (or 8 out of 8 for vendors who did not participate in the Linux portion of the evaluation) included at least one correct block.
MITRE Round 4 - Combined Prevention Test and Detection results
While a number of vendors claim results of 100% protection in the MITRE evaluation, their evaluation results for blocking malicious activity are not equal, and the differences between them can have a significant impact on an affected organization. We encourage you to take a close look to understand when in the attack sequence the protections occurred, knowing that blocking earlier is better.
MITRE Protection Evaluation
The protection portion of the evaluation included a total of nine tests, each of which involved a number of attack substeps. In order to obtain a successful result for each test, vendors were required to block at least one of the attack substeps, and were prohibited from blocking actions which were noted as valid actions. Failure to block at least one substep would result in failure of that test, and blocking an action that was noted as valid by MITRE disqualified the vendor's results for that step.
Each test result also includes specific details in the sequence where the prevention took place. This additional data is critical in enabling security analysts to gauge how effective each vendor is at preventing malicious activities and minimizing impact to the organization. This detail also highlights where in each attack sequence prevention actually occurred, and enables analysts to more accurately determine which vendor(s) were most effective at handling the attack and minimizing impact; this also illustrates significant differences in overall efficacy, even among vendors claiming "100% coverage".
To illustrate this concept, we will explore two of these test scenarios, and compare the MITRE Round 4 performance of three vendors (CrowdStrike, Microsoft and Palo Alto Networks), all of whom successfully completed this test, and all of whom attained a result of 9 out of 9 or ‘100%’ for the overall protection evaluation.
Protection Test #4
The scenario for test #4 is described by MITRE as follows: “Compromise Domain Host”. This test contains a total of 2 steps and is the simplest test in the protection evaluation when measured by the total number of steps. This step utilizes native windows utilities found within the operating system, and uses valid functions within those utilities to perform these actions. This is a “living-off-the-land” attack sequence which involves no external code. There are no unblockable steps in this test; both of the steps are valid block points. The summary of actions which occur are as follows:
- For the first phase (Step 1), the attackers use vssadmin.exe (the Windows Volume Shadow Copy Service) to make a snapshot of the C:\ drive which contains the NTDS user database. This enables the attackers to access the database while avoiding many protections. NTDS contains critical user information including usernames, group membership and password hashes; all of which are valuable to attackers.
- For the second phase (Step 2), the Attackers use reg.exe to export the HKEY_LOCAL_MACHINE\SAM Registry Hive, which contains additional local user account information and password hashes which are also useful to the attackers
While this is a short sequence, each step can provide immediate impact and value for the attacker, since each step yields highly valuable information.
For test #4, Palo Alto Networks prevented the attack sequence at the first step, denying the attacker any data of value that would have otherwise been gleaned. By terminating the hostile process tree, the subsequent action did not take place, and no data was gained by the attacker.
When we compare this result to that of a different vendor, in this case Microsoft, we see that in the evaluation they achieved prevention on the SECOND step of test #4, but allowed the first step to complete which permitted the attacker to create a volume shadow copy of the C:\ volume and thereby obtain a copy of the ntds.dit database it contained. NTDS contains user account information, including account names, as well as group membership. Most critically, it also stores the password hashes (encrypted text) for each account.
| Technique/Substep | Blockable | Microsoft Blocked | Palo Alto Networks Blocked |
|---|---|---|---|
| OS Credential Dumping: NTDS | YES | NO | YES |
| OS Credential Dumping: Security Account Manager | YES | YES |
What is the Impact?
Obtaining these hashes enables an attacker to either crack the hashes at their leisure via brute-force means or employ the hashes directly via pass-the-hash techniques, wherein the hashed value of a password can simply be substituted for the password itself. Armed with this data, an attacker can know which accounts to target (for example, privileged accounts or application service accounts) and can gain additional high value access with this information. Therefore, even though the attack sequence was ultimately prevented, with Microsoft Defender, prevention did not occur in time to stop the simulated attacker from gaining access to critical information, and the prevention that does ultimately occur only stops further actions from taking place via this attack chain, and does not address the information that attacker had already gained.
Protection Test #8
The scenario for test #8 is a much more extensive compromise attempt of a target domain host, utilizing both keylogging and browser credential harvesting methods in conjunction with an attacker-installed command and control (c2) backdoor for use in issuing instructions to the endpoint and to facilitate secure data exfiltration. This sequence contains a total of 21 steps and is the largest test by number of steps in the entire prevention evaluation. The summary of actions which occur in this test sequence are as follows:
- For the first phase (Steps 1 and 2), the attackers use stolen credentials to gain access to the target endpoint via the admin$ share. NOTE: Because these activities use a valid account and make a standard SMB connection to a windows share, the steps in this phase are NOT considered a valid block points in the MITRE test. Blocking the actions at this stage will result in “a failure to properly assess” notation from the MITRE Evaluator, with neither a success nor a failure assigned.
- For the second phase (Steps 3 - 7), the Attacker copies over an instance of the exaramel malware utility, and configures that utility to run as a new Windows Service called “Windows Check AV”. The Malware is configured to run automatically on windows startup and the newly created malware service is started by the attacker. NOTE: This Sequence and all which follow are considered valid block points by MITRE.
- For the third phase (Steps 8-10), the attacker installs and loads a network proxy dll which is used to set up and maintain an encrypted channel between the attacker’s host and the victim host. This channel will be used in later exfiltrations.
- For the fourth Phase (Steps 11 - 17), the attacker performs an extensive series of data collection activities in an effort to gain additional information about the endpoint and its configuration, as well as additional user account information from browser ID caches, and via use of keystroke capture. Additional tools to facilitate the collection and reading of browser databases (oradump.exe) and capture keystrokes (mslog.exe) are deployed and used to support these activities.
- In the final phase of the attack sequence (Steps 18-21), the attacker exfiltrates the data collected previously via the encrypted channel set up earlier. The oradump.exe and mslog.exe instances used for collection are deleted, along with any local copies of collected data. The installed exaramel services remains present and active on the victim.
This attack totals 21 steps from beginning to end. The first 2 steps in this test are considered benign and not valid for blocking by MITRE. The remaining 19 steps are all considered valid block points for this test and a block that happens on any of these remaining steps will yield a pass for that test.
In test #8, Palo Alto Networks detected every step in the attack sequence up to the point of prevention. Prevention occurred on step 3 in the sequence (the first blockable step). As a result, the endpoint remained unimpacted by the attack sequence, and no further remediation actions are required to address elements of this attack
Conversely, if we look at the performance of a different vendor, namely CrowdStrike, we see that while their product detected each step in the test sequence up to the point of prevention (process kill and binary quarantine). Prevention occurred at step 7 in the attack sequence (the 5th blockable step).
| Technique/Substep | Blockable | CrowdStrike Blocked | Palo Alto Networks Blocked |
|---|---|---|---|
| Valid Accounts: Domain Accounts | NO | N/A | N/A |
| Remote Services: SMB/Windows Admin Shares | NO | N/A | N/A |
| Ingress Tool Transfer (wsmproav.exe) - exaramel malware | YES | NO | YES |
| Ingress Tool Transfer (service binary) - exaramel malware | YES | NO | |
| Create or Modify System Process: Windows Service - exaramel malware | YES | NO | |
| Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder | YES | NO | |
| System Services: Service Execution | YES | YES | |
| Remaining Steps (8 - 21) | YES |
As a consequence, the test system suffered the following impacts before the attack chain was ultimately prevented:
- The attackers were able to successfully transfer the exaramel backdoor malware to the target
- The exaramel malware was registered as a service
- Exaramel was configured to start automatically on the next reboot.
In summary, despite preventing the attack from completing, the CrowdStrike response for this test left a backdoor fully installed and configured to start automatically upon a future reboot. This leaves the system in a compromised state and at risk of suffering future impact by the attackers unless it is promptly remediated by analysts.
In order to fully remediate this attack, not only must the hostile process chain be terminated (it eventually was), but now the changes that were made by the attacker must also be discovered and corrected. In scenarios where the attack is widespread across an organization, the full extent of these remediations must be discovered and corrected at significant scale. While prevention did ultimately stop this attack chain from fully completing, it left numerous hostile artifacts and potentially damaging changes within the environment that must be separately addressed in order to achieve full remediation.
What is the Impact?
This attack sequence involves the installation and enabling of a number of attacker utilities, including a command and control module, a keylogger, and the use of a utility to dump and read the contents of various local database files, which are then utilized to collect information vital to the attackers and then exfiltrate that information to an external destination. Throughout the course of the attack, successive degrees of impact occur as these tools are installed, activated and then employed. While a delay in prevention may disrupt the attack before it can fully complete, any significant delay, as was observed in the CrowdStrike result, leaves the machine in a compromised state, and potentially open to further actions by the attackers via the elements which were installed and configured ahead of the prevention action. Early and accurate prevention is critical in minimizing this impact and in reducing the overall recovery efforts undertaken by the response team.
In Summary
These collective steps described in these examples represent a significant number of security impacts that will occur if each are allowed to complete, and each step that is completed results in negative impact to the organization that must ultimately be remediated and may include business disruption, financial losses and other business impacts. Therefore it is critical that the security tool not only block the attack, but block this attack as early as possible in the attack sequence to minimize the impact to the organization. This is why it becomes important to gauge the point at which each vendor blocks the attack; the earlier the better. When we apply this lens to different vendors, we can quickly see that they perform very differently, and those differences carry significant implications for the security team.
MITRE Round 4 - Combined Prevention Efficacy and Detection Results
Chart 2: Y-Axis tracks out-of-the-box Technique Detections (config changes removed). X-Axis tracks substep level prevention results (97 blockable substeps).
What This Means for Buyers
These examples highlight the importance of not just IF prevention happens, but WHEN prevention happens during an attack; delays have significant consequences. We will be posting additional blogs in the coming weeks to analyze the MITRE results and what they mean and how you can use the results to better understand the efficacy of different security solutions.
Learn more about how to unpack the MITRE results by joining our ‘Dissecting the 2022 MITRE ATT&CK Evaluations Webinar and visiting our MITRE webpage.