Significant cyber incident

HSE Conti ransomware attack

In May 2021 Conti ransomware hit Ireland's Health Service Executive (HSE), the public health service.

In short

In May 2021 Conti ransomware hit Ireland's Health Service Executive (HSE), the public health service. A staff member opened a malicious Excel file in a phishing email, and the first workstation was infected on 18 March 2021. The ransomware ran on 14 May 2021. The HSE shut down its whole IT network, hospitals fell back to paper, and the HSE says it did not pay the ransom.

What happened

On 14 May 2021 the Health Service Executive (HSE), which runs public health services in Ireland, was hit by ransomware. The National Cyber Security Centre (NCSC) said at the time that it was likely carried out by criminal actors and that initial reports pointed to a human-operated Conti ransomware attack. The HSE shut down all of its IT systems as a precaution. The NCSC said some non-emergency procedures were postponed and that the national vaccination program was not affected.

The attack did not start on 14 May. The independent review by PwC says a staff member opened a malicious Excel file attached to a phishing email, and the first workstation was infected on 18 March 2021. Antivirus detected Cobalt Strike and Mimikatz on that workstation on 31 March, but it was in monitor mode and did not block them. The attackers first compromised HSE servers between 7 and 12 May 2021, and then systems in six voluntary hospitals and one statutory hospital.

The Department of Health network was also reached. The Department says initial access was on 11 May 2021. The NCSC said an attempt to run ransomware there on 14 May was detected and stopped, and that the attacks were believed to be part of the same campaign targeting the Irish health sector.

The HSE says the aim of the attack was to disrupt its services, copy information and demand a ransom, and that no ransom was paid. The Comptroller and Auditor General reports that a demand of almost 20 million US dollars was received. On 21 May 2021 the attacker posted a link to a decryption tool. Recovery took months: the Comptroller and Auditor General says full access to systems was not restored until late September 2021.

Timeline

  1. 18 March 2021

    A HSE staff member opens a malicious Excel file attached to a phishing email. The first workstation is infected.

  2. 31 March 2021

    Antivirus detects Cobalt Strike and Mimikatz on that workstation. It is in monitor mode and does not block them.

  3. 7 to 12 May 2021

    The attacker compromises HSE servers for the first time, then systems in six voluntary hospitals and one statutory hospital.

  4. 11 May 2021

    Initial access to the Department of Health network, according to the Department.

  5. 14 May 2021

    Conti ransomware runs on HSE systems. The HSE shuts down all IT systems. The NCSC issues an alert. A ransomware attempt on the Department of Health is detected and stopped.

  6. 15 May 2021

    The HSE reports the breach to the Data Protection Commission.

  7. 16 May 2021

    The NCSC updates its alert with analysis, indicators and ATT&CK techniques.

  8. 20 May 2021

    The HSE obtains a court order restricting the sharing of its data.

  9. 21 May 2021

    The attacker posts a link to a decryption tool.

  10. 24 May 2021

    A decryption process starts to allow secure recovery of systems.

  11. 21 September 2021

    All servers are considered decrypted and 99% of applications restored, 130 days after the encryption.

  12. November 2021

    The HSE Board accepts the PwC recommendations.

  13. December 2021

    The independent post incident review by PwC is published.

  14. End of December 2021

    Gardai give the HSE a copy of the copied documents under a mutual legal assistance treaty.

How the attack worked

Two official accounts describe the attack: the PwC post incident review commissioned by the HSE, and the NCSC alert of 14 May 2021 with its update of 16 May. The Comptroller and Auditor General repeats parts of both. The sources below do not name the people behind the attack.

Getting in

The PwC review says a staff member interacted with a malicious Microsoft Office Excel file attached to a phishing email on 18 March 2021. This infected the first workstation. Phishing (T1566) is the matching ATT&CK technique on attack.mitre.org; the review does not give an ID.

Staying hidden

On 31 March 2021 antivirus detected Cobalt Strike and Mimikatz on that workstation. The antivirus was in monitor mode, so it did not block the commands. The NCSC says Cobalt Strike beacons on systems suggest lateral movement before the ransomware payload ran. The NCSC also told defenders to watch for precursor malware that may have come before ransomware.

Spreading

The attacker first compromised HSE servers between 7 and 12 May 2021, then systems in six voluntary hospitals and one statutory hospital. The Comptroller and Auditor General says trust relationships between the HSE and the Department of Health, and between the HSE and Beaumont Hospital, helped the attackers reach those networks. The NCSC found that psexec was used with compromised user credentials to run the payload on endpoints, and that batch files copied the malware to endpoints.

The ransomware

The NCSC found Conti ransomware version 3. It used WMIC to delete shadow copies and renamed encrypted files with a .FEEDC extension. The NCSC listed these ATT&CK techniques: Windows Management Instrumentation (T1047), Data Encrypted for Impact (T1486) and Inhibit System Recovery (T1490). The PwC review found over 2,800 servers and 3,500 workstations with evidence of encryption, across 15 domains.

How the attack ran
  1. Phishing emailA staff member opens a malicious Excel file attached to a phishing email on 18 March 2021Technique T1566 (MITRE ATT&CK, opens in a new tab)
  2. Tools placedCobalt Strike and Mimikatz are detected on the workstation on 31 March, but antivirus only monitors
  3. SpreadServers and hospital systems are compromised between 7 and 12 May, with psexec and compromised credentials
  4. Shadow copies deletedWMIC is used to delete shadow copiesTechnique T1490 (MITRE ATT&CK, opens in a new tab)
  5. Ransomware runsConti encrypts files on 14 May 2021 and renames them .FEEDCTechnique T1486 (MITRE ATT&CK, opens in a new tab)

Impact

Systems: the HSE had over 70,000 IT devices in about 4,000 locations and shut down its network. The PwC review found over 2,800 servers and 3,500 workstations with evidence of encryption. HSE management reported that 80% of its environment was encrypted. The Comptroller and Auditor General says all servers were considered decrypted by 21 September 2021, 130 days after the encryption.

Patients: radiation treatment ceased in five centers in the days after the attack. Staff used manual workarounds and paper records. The HSE reports that some data is irretrievable.

Data: the HSE says a small amount of its information appeared on the dark web and was taken down, and that it has no evidence the information was used in scams or fraud. It notified 90,000 people.

Cost: in 2021 the HSE spent 37 million euro on revenue costs and 14 million euro on capital costs because of the attack. Legal costs were 2.6 million euro to July 2022. The Comptroller and Auditor General says the full cost has not been quantified. The initial HSE estimate for the PwC recommendations is almost 657 million euro over seven years.

Other bodies: the Department of Health had direct costs of almost 647,700 euro.

Government and regulator response

  • 14 May 2021: the NCSC activated its crisis response procedures to support the HSE and published an alert. The HSE shut down all IT systems.
  • 15 May 2021: the HSE reported the breach to the Data Protection Commission.
  • 20 May 2021: the HSE obtained a court order restricting the sharing of its data.
  • 21 to 24 May 2021: the attacker posted a link to a decryption tool and a decryption process started.
  • Gardai worked with international law enforcement on the investigation. At the end of December 2021 they gave the HSE a copy of the copied documents under a mutual legal assistance treaty.
  • The NCSC told defenders to isolate domain controllers, block egress to the internet, protect privileged accounts and harden endpoints.
  • November 2021: the HSE Board accepted the PwC recommendations. The PwC report has 83 key recommendations.
  • After the attack the HSE put in place an enhanced 24-hour network monitoring service (Comptroller and Auditor General).

Lessons

  1. Alerts must be acted on

    Antivirus detected Cobalt Strike and Mimikatz on 31 March 2021 but was in monitor mode and did not block them. The attack ran for six more weeks. A detection nobody acts on gives no protection.

  2. Someone must own cyber security

    The PwC review found no single responsible owner for cybersecurity at senior executive or management level. Name an owner with authority and budget.

  3. Plan the recovery, not only the defense

    There were no documented incident response runbooks or IT recovery plans for a wide-scale ransomware event. Write and rehearse them.

  4. Old systems are a standing risk

    The HSE had over 30,000 Windows 7 workstations and a 2018 audit had found outdated server software. Unsupported systems need a replacement date.

  5. Trust links spread the damage

    Trust relationships between the HSE, the Department of Health and Beaumont Hospital helped the attackers reach them. Good monitoring at the Department and Beaumont stopped widespread encryption there.

  6. Monitoring hours matter

    Before the attack the antivirus monitoring service ran from 8 am to 6 pm with on-call cover after. Ransomware often runs out of office hours, so cover must be around the clock.

What to check in your own organization

  1. Antivirus mode

    Confirm endpoint protection is set to block, not only monitor, on every device. Review any alerts about Cobalt Strike or Mimikatz from the past year.

  2. Phishing with Office files

    Check how email filtering handles Office attachments and how staff report suspicious mail.

  3. Unsupported systems

    List all systems that are out of vendor support, such as old Windows versions, with a date to replace each.

  4. Trust between networks

    Review trust links between your network and partner or sub-organization networks. Remove any that are not needed.

  5. Monitoring hours

    Check that alerts are watched 24 hours a day, not only in office hours.

  6. Recovery plans

    Check you have written runbooks for a ransomware event and that offline backups were restore-tested.

  7. Named owner

    Confirm a single named person owns cyber security and reports to the board.

Questions

Who attacked the HSE?

The NCSC said the attack was likely by criminal actors and that initial reports indicated Conti ransomware. The Irish sources opened do not name a group beyond that.

Did the HSE pay the ransom?

The HSE says no ransom was paid. The Comptroller and Auditor General reports a demand of almost 20 million US dollars and says the HSE did not engage with the attackers.

How long did recovery take?

The Comptroller and Auditor General says full access to systems was not restored until late September 2021. All servers were considered decrypted by 21 September 2021, 130 days after the encryption.

Was patient data stolen?

The HSE says the attackers accessed and copied information. It notified 90,000 people. It says a small amount of information appeared on the dark web and was taken down, and it has no evidence of use in scams or fraud.

Was the Department of Health affected?

Yes. The Department says initial access was on 11 May 2021. The NCSC said a ransomware attempt on 14 May was detected and stopped.

What did the independent review find?

PwC found no single owner for cyber security, no documented ransomware runbooks or recovery plans, and over 30,000 Windows 7 workstations. It made 83 key recommendations, which the HSE Board accepted in November 2021.

Sources

  1. Conti cyber attack on the HSE: Independent post incident review (PDF) Health Service Executive (report by PwC)
  2. Conti cyber attack on the HSE: Independent post incident review (publication page) Health Service Executive
  3. NCSC Alert: Ransomware Attack on HSE Network, 14 May 2021 National Cyber Security Centre (Ireland)
  4. NCSC Alert: Ransomware Attack on Health Sector, update of 16 May 2021 National Cyber Security Centre (Ireland)
  5. Report on the Accounts of the Public Services 2021, chapter 12: Financial impact of cyber security attack Office of the Comptroller and Auditor General
  6. Cyber-attack and HSE response Health Service Executive