The question after containment is whether you trust it
Once containment holds and the encryption has stopped spreading, the work changes character. The question is no longer what is happening. It becomes which parts of this environment you are prepared to hand back to the business, and that is a different standard from clean. A host where the endpoint tools report nothing has not been proven clean. It has been searched with whatever coverage you happen to have, and the result carries exactly as much weight as that coverage does.
Three positions are available. The first is to disinfect what you have and reconnect it. The second is to rebuild the hosts and identities where the evidence puts the attacker, and reconnect everything else. The third is to build from known-good and restore data into it. The gap between the first and the third is an order of magnitude in hours, disruption and cost, which is exactly why the decision gets made on cost when it should be made on what the evidence will support.
Cleaning is defensible when two things are true together: the entry path is explained, and the blast radius is bounded by data you still hold. Where the entry path is unexplained, disinfecting a host is a wager that the persistence you found is all the persistence there was, placed under fatigue by the same team whose telemetry missed the intrusion in the first place. That wager fails in a predictable shape. The environment comes back, trades for a fortnight, and is encrypted a second time by an operator who never left.
Which conditions force a rebuild?
Work the table by marking every row you can observe in your own environment. The rows are not weighted equally and they do not average out. A single row in the lower half forces the outcome in its right-hand column no matter how many rows above it look reassuring, because those rows describe exposure of credential material or destruction of the record, and neither is undone by cleaning a disk.
The absence of findings does not license cleaning. Most of these conditions are observable only if you had the telemetry to observe them, so an environment with ninety days of endpoint detection data and one with a fortnight of firewall logs are answering different questions when they both report nothing found. Where you cannot test a row, treat it as unknown and let the unknown push the decision toward rebuild for that tier. Write down which rows you could not test, because that list is the honest scope of the recovery and somebody will ask for it later.
| Observed condition | What it tells you | Recovery position it supports |
|---|---|---|
| Entry vector identified, activity confined to one host, no credential access observed | The blast radius is bounded and provable from data you hold | Clean in place, then watch that host for a defined period |
| Credential theft tooling ran anywhere, or process memory on a Windows host was read | Every credential cached on that host is compromised, including accounts that signed in afterwards | Rebuild the host and rotate every identity that touched it |
| A writable domain controller was compromised or the directory database was copied | All domain credential material is exposed, including key material users never see | Full identity plane rebuild, whatever you decide about the servers |
| A domain administrator or global administrator account was used by the attacker | Everything that account could reach is in scope, which in most estates is everything | Full identity rebuild, selective host rebuild driven by evidence |
| Persistence found in more than one form on the same host | The operator planned for eviction, so the count you have is a floor and the total is unknowable | Rebuild that host and treat the whole segment as suspect |
| Event logs cleared, telemetry gaps, or timestamps that do not reconcile | You cannot prove what was untouched, so a clean result carries no weight | Rebuild whatever the missing period covered |
| Backups encrypted or deleted, or the backup service account was used | The known-good you intended to rebuild from may not be known-good | Prove backup integrity before restoring, or build from installation media |
| Entry vector still unexplained when forensics runs out of road | Cleaning becomes a bet on a number of implants you have no way to estimate | Rebuild the affected tier and keep the old environment isolated for analysis |
| Regulated or sensitive data in scope, or the breach is notifiable | The recovery will be read later by an insurer, a regulator or a court | Either path is available, but the evidence has to be preserved before you destroy it |
| Business pressure to be trading again by a fixed date | Time pressure can change the sequence in which systems return; it cannot change what has to be proven first | Stage the reconnection so revenue systems return first, under heavier monitoring |
Observed conditions after containment, and the recovery position each one supports
What a domain controller compromise forces you to do
When a writable domain controller is compromised, or its database is copied out, every secret the domain holds goes with it. That is broader than the password hashes. It includes the key material that Kerberos authentication chains up to, and resetting every user password in the directory does nothing about it. This is the row in the table above that gets underestimated most often, because the visible damage may be one encrypted file server while the actual exposure is the whole authentication system.
The krbtgt account is the specific problem. Its key signs and encrypts every ticket-granting ticket in the domain, so an attacker holding it can mint tickets for any principal. Microsoft's Active Directory forest recovery guidance is unambiguous about the remedy: reset the krbtgt password twice, waiting ten hours between the two resets. Ten hours is the default maximum lifetime for both user and service tickets, so the wait lets tickets issued under the old key expire on their own instead of breaking mid-session, and where a domain is configured with a longer lifetime the wait has to exceed that configured value. Two resets are needed because the password history for that account holds two passwords, so a single reset leaves the previous key in history and forged tickets built on it keep validating. Running both resets back to back with no wait produces an authentication and replication outage of your own making, which is how a recovery generates its own second incident. Budget for the disruption either way, because every ticket-granting ticket already issued stops being accepted and the dependable way to return a machine to service is a reboot.
Group managed service accounts need separate handling. Their passwords derive from the Key Distribution Service root key, so an attacker who read that root key object, together with an account's security identifier and its managed password identifier, can compute those service account passwords offline and keep computing them. Rolling the accounts changes nothing while the root key stays the same. Microsoft documents the recovery: create a new root key, get every group managed service account onto it by letting the passwords roll or by recreating the accounts outright, then delete the old root key object once they have all moved. A new root key is not usable in production for about ten hours after creation, so this belongs in the plan from the first day of the recovery.
The same reasoning covers the domain controller computer account and the trust passwords on each side of every domain trust, both of which the forest recovery procedure resets when the recovery is driven by a security breach. If the identity plane is in scope, rebuilding the file server is not the job. The identity plane sets the sequence for everything else, and until it is rebuilt or fully rotated, nothing you reconnect to it is trustworthy no matter how carefully that system was rebuilt.
What order do you rotate credentials in?
Credential rotation has a dependency problem. Every rotation is performed using access that may itself be compromised, and several of the rotations will lock out the people performing them. Before anything else, establish a clean administrative path: a freshly built device that has never touched the compromised environment, a cloud-only break-glass account with phishing-resistant multi-factor authentication enrolled on it, credentials held offline, and an alert that fires whenever that account signs in. Everything below is performed from there.
The ordering rule is to rotate the things that issue trust before the things that consume it, so the key material behind Kerberos goes before the service accounts that authenticate with it, and federation certificates go before the applications that trust them. Reverse that order and you rotate downstream credentials the attacker re-harvests using the upstream access you left in place, which buys a long night of work and no change in position.
The second rule is that a reset and a session revocation are one operation. A password change does not retire an existing session, and in most estates what the attacker holds is a live session, which no password change touches. The table sets out the order. The fourth column is the part worth arguing about internally, because that is where an item gets quietly missed.
| Order | What gets rotated | Does a password reset close it | How it gets missed |
|---|---|---|---|
| 1 | Break-glass administrative accounts on a clean device | Yes, once phishing-resistant multi-factor authentication is re-enrolled | Responders rotate everything else first and lock themselves out of their own tenant |
| 2 | Privileged directory accounts, domain and global administrators | Only when sessions and refresh tokens are revoked in the same action | The reset lands and the attacker's existing session keeps working |
| 3 | The krbtgt account, twice, ten hours apart | Not applicable, the reset is of key material used to sign tickets | One reset is performed, so the old key stays in history and forged tickets still validate |
| 4 | Domain controller computer account and trust passwords | Handled by the forest recovery procedure, not by a user password reset | Treated as routine maintenance and deferred until after reconnection |
| 5 | Key Distribution Service root key and group managed service accounts | No, those passwords derive from the root key | Service accounts keep deriving passwords from a root key the attacker already read |
| 6 | Federation and token signing certificates | No, these are certificates and get replaced, so no reset touches them | Only one replacement certificate is issued, leaving the old one able to sign valid tokens |
| 7 | Service accounts, scheduled task identities and application pool identities | Yes, and the service fails unless the new password reaches everywhere it is stored | Rotation is deferred because nobody can say what the account actually runs |
| 8 | Remote access, virtual private network, jump hosts and management interfaces | Only with device certificates and appliance local accounts reissued as well | The appliance local administrator is not in the directory, so no reset script touches it |
| 9 | Client secrets and certificates on application registrations | No, an application credential is independent of every user password | An application with mail permissions keeps its access after every human account is reset |
| 10 | Consent grants and service principals with directory permissions | No, a granted permission survives until the grant itself is revoked | Consent given months earlier is never enumerated and outlives the entire rebuild |
| 11 | Keys held by partners, payment providers and data platforms | No, these live outside your directory entirely | No inventory exists, so the keys surface one at a time as things break |
| 12 | Code signing keys and build pipeline credentials | No, and a stolen signing key keeps its value long after the incident closes | Rebuilt hosts are configured to trust artefacts signed with a compromised key |
| 13 | Mail flow rules, forwarding addresses and delegate permissions | No, rules and delegations are configuration, so no credential change removes them | The mailbox password is new and the mailbox is still copying every invoice to the attacker |
Re-trust order, whether a password reset actually closes the item, and how each one gets missed
What survives a password reset?
Refresh tokens and browser session cookies are the first surprise. In Microsoft Entra ID, invalidating them is an action separate from the reset: Revoke sessions on the user in the admin centre, or revokeSignInSessions through the Graph API, which stamps a validity date on the account and forces every application to obtain new tokens. Two details change how you use it. A few minutes can pass before the revocation takes effect, so a revocation issued in the same breath as a reset has to be verified afterwards. And it does not revoke sign-in sessions for external users, because those users authenticate in their home tenant, which makes every guest holding access to your data a separate piece of work.
Access tokens already issued are the second gap. A standard access token has a default lifetime in the region of an hour, so an attacker holding a live one keeps that access for the remainder of its life after the reset lands. Where both the client and the resource support continuous access evaluation, token lifetimes extend to as much as twenty-eight hours, and in exchange revocation on critical events such as a password change happens close to real time, though propagation can still take up to about fifteen minutes. The instruction is the same in both cases. Reset, revoke sessions, then read the sign-in logs for that account and watch what the attacker's client does next.
Application identity is the gap that most often survives an entire recovery. A consent grant made months ago, a service principal carrying directory permissions, a client secret uploaded to an application registration: none of these are touched by resetting human passwords, and none appear in the account lists most teams work from. Enumerate every application with permissions over mail, files or the directory, rotate its secrets and certificates, and remove anything nobody present can explain.
When is it safe to reconnect a rebuilt system?
Rebuilt systems should not be returned to the network they were compromised on. Build a new segment with new addressing and its own controls, and let systems enter it only once they have been rebuilt or verified. Nothing crosses from the old environment except data that has been through a defined check, and the old environment stays isolated and powered down instead of decommissioned, because you are going to want it later. Reconnection then becomes a sequence of gates, each carrying a criterion written before the stage starts, so the decision to proceed is measured against something agreed while everyone was calm.
The monitoring goes in before the traffic does. Endpoint detection is installed and reporting on a host before that host receives an address. Egress filtering denies by default, with an allow-list you build deliberately, because an allow-list generated from the first day of live traffic encodes whatever the attacker is still doing. Name resolution is logged in full, and sign-in alerting runs on the rebuilt identity plane with a specific alert on the break-glass accounts. Without that visibility there is no record of why each system was let back in, and an insurer reviewing the claim will ask for exactly that record.
- Identity goes first: the directory is rebuilt or fully rotated, break-glass access is proven from a clean device, and administrative access to the new segment is possible only from purpose-built workstations.
- Core services follow on rebuilt hosts, under service accounts created new for the purpose, since a restored service account carries a password the attacker may already hold.
- Data-bearing systems come next, restored from a backup whose restore point sits before the earliest confirmed attacker activity, and scanned offline before the volume is mounted and reachable.
- User endpoints are rebuilt from a known image, and multi-factor authentication is re-enrolled as each owner returns, because carrying the old enrolment across also carries whatever the attacker registered.
- Internet-facing services return only after the entry vector is patched, verified by someone other than the person who patched it, and re-tested from outside the network.
- Third-party and partner integrations are last, after the keys in the rotation table have been reissued and each partner has confirmed the new ones are in use.
Preserving evidence while you rebuild
Rebuilding destroys the evidence. Wiping and reimaging is the correct remediation for a compromised host, and it is also the most common way an Australian business loses the ability to answer questions it will be asked for the next two years. The fix is unglamorous. Image before you wipe, and hold the images on media the rebuild cannot reach.
The people who will ask are known in advance. A cyber policy commonly requires the insurer's agreement to the forensic provider and a forensic account of what happened before a claim is settled, so decisions taken in week one shape what eventually gets paid. The Notifiable Data Breaches scheme under the Privacy Act 1988 requires you to assess a suspected eligible data breach expeditiously and in any event within thirty days, and you cannot assess what you deleted on day three. Cybercrime is reported through ReportCyber, run by the Australian Signals Directorate's Australian Cyber Security Centre, and critical infrastructure entities carry separate reporting duties under the Security of Critical Infrastructure Act. A civil claim years later is decided on records, and the records will be whatever you kept in the first fortnight.
Capture full disk images of the first host you can attribute to the attacker and of every host where credential access is suspected. Take volatile memory where the machine is still running and the decision is made early enough to matter. Export the logs that age out fastest, usually identity sign-in logs, endpoint telemetry, firewall and proxy records and mailbox audit data, because retention windows will quietly delete the period you care about while the recovery is still running. Keep the ransom note and every communication with the operator. Keep a contemporaneous action log of what was changed, by whom and at what time, which is separately useful when you need to show that a given change was yours and not theirs.
Settle the engagement structure on day one. A forensic report commissioned directly by the business is ordinarily discoverable. A report commissioned by external counsel for the dominant purpose of providing legal advice may attract legal professional privilege, and Australian courts have examined those arrangements closely enough that the outcome turns on how the engagement genuinely operated and not on what the cover page says. That is a decision for your lawyer, and it has to be made before the first forensic invoice is raised.
What has to be true before anyone declares normal operation?
The end of an incident is a decision, and it should be made against criteria written down before the recovery started. Left undefined, it gets made by exhaustion, usually by whoever is least willing to keep saying no to the operations manager.
Give every criterion a named owner and a written confirmation. A recovery of any size is worked by internal staff, a managed service provider, a forensic firm and at least one software vendor at once, and each assumes somebody else covered the item that turns out to be missed. These are the criteria worth agreeing in advance.
- The entry vector is named in a sentence, the fix is deployed everywhere the same weakness exists, and someone independent has re-tested it from outside.
- Every row of the credential rotation table has a named owner who has confirmed in writing that the rotation is complete, including the rows that live outside the directory.
- No detections attributable to the actor across rebuilt systems for a watch period set against the dwell time, so a six-week intrusion is not signed off after three quiet days.
- Privileged accounts hold phishing-resistant multi-factor authentication, legacy authentication protocols are disabled, and the consent and application registration inventory has actually been reviewed.
- Detection coverage exists for the specific techniques observed in this incident, and someone has fired a test alert through the full path to a human who was expecting it.
- Backups are running in the new environment and a restore has been performed from them, because the backup regime that failed you is not the one to carry forward untested.
- Notification and reporting obligations are either closed out or tracked on a timetable with named owners and dates.
- The old environment and the evidence set are retained, isolated, and access to them is recorded.
Why environments get hit again a month later
The second incident follows a consistent shape. The environment is restored, the business resumes trading, and within a few weeks the same operator returns, often with better knowledge of the estate than they had the first time. The review almost always finds the same cause. The entry vector was never actually closed, or it was closed on the systems that were rebuilt and left open on one that was not.
There is one question to put to every system before it reconnects, and it is not whether the scan came back clean. Ask how you know this specific path is closed, and what you would expect to see if it were not. An answer that amounts to the tools being quiet is the same answer the organisation had the week before the first encryption. The paths below are the ones that keep turning up, and they are short enough to work through as a checklist.
- An internet-facing appliance was patched but never examined for the implant it was already carrying.
- A remote access account without multi-factor authentication was missed by the rotation because it did not exist in the directory.
- A local administrator password reused across the estate was changed only on the machines that were rebuilt.
- A backup console or hypervisor kept its own local accounts, which no directory-wide reset reaches.
- An application consent grant was never enumerated, so the application held its access straight through the rebuild.
- A mailbox rule survived every reset and carried on forwarding to an external address.
- A scheduled task came back with a server restored from a backup taken after the attacker was already inside, which is why a restore point is chosen against the intrusion timeline and never by taking the most recent copy.
How we work a remediation
Black Shard runs incident response and breach remediation for Australian businesses, and the decision described here is the one we make jointly with a client and put in writing. That record covers the conditions observed, the recovery position they support, what was rotated and in what order, what evidence was preserved before anything was destroyed, and the criteria met at each reconnection gate. It exists because the insurer, the regulator, the board and possibly a court will each read a version of that account later, and reconstructing it from memory months afterwards is how a defensible recovery turns into an indefensible one.
The work that follows sits across the rest of what we do. Rebuilding an identity plane is Entra ID and Azure security work. Proving the entry vector is closed is penetration testing, and a repeat vulnerability scan does not answer it. Watching a reconnected estate for the return of an operator is attack surface monitoring. Holding the standard steady while the business pushes to be back up is the part a vCISO carries. Our approach and trust pages set out how an engagement runs, and the breach page is the direct path if you are working an incident now.
