Read on Twitter
I was at Mothercare, sent out a warning when I saw the patch, but, IT management decided to ignore it (never deployed patches to stores).
We were hit bad. I had to create my own tools to deploy the patch & scan for coverage.
Contd. https://twitter.com/todayininfosec/status/1197757151034273798
We were hit bad. I had to create my own tools to deploy the patch & scan for coverage.
Contd. https://twitter.com/todayininfosec/status/1197757151034273798
The store servers (win2k3) & tills (XPe) were all cloned from a single image. Same username+pass. The Comms circuits terminated at head office, there was no IDS/IPS, or even ACLs to prevent inter store traffic.
Perfect setup for a worm.
Contd.
Perfect setup for a worm.
Contd.
The tills were very low powered and very slow. Easily 15mins to reboot. Past EoL, but retail doesn't like to spend money on stuff it doesn't have to.
Store infrastructure was managed by an MSP, they were out of their depth now, plus the DDoS element meant they couldn't.
Contd.
Store infrastructure was managed by an MSP, they were out of their depth now, plus the DDoS element meant they couldn't.
Contd.
Due to the low powered tills the end point security policy was massively ineffective and did nothing to protect.
The stores had to 'open drawer trade' due to comms being DDoS'd, meaning no credit card auth.
Sales were down massively.
Contd
The stores had to 'open drawer trade' due to comms being DDoS'd, meaning no credit card auth.
Sales were down massively.
Contd
We had no monitoring in place for the stores, that was the MSP's area. Had to wait to be called by stores to find out this was hitting them.
Working with our Comms guy I found it was Conficker. He put rules in to limit the interstore DDoS traffic.
Contd.
Working with our Comms guy I found it was Conficker. He put rules in to limit the interstore DDoS traffic.
Contd.
The rules were effective enough that stores could trade. But, everything was still there attempting to DDoS & infect.
I was just about to fly out to a family wedding. I was begged not to go! I said that if things were still like this when I got back I'd step in.
Contd.
I was just about to fly out to a family wedding. I was begged not to go! I said that if things were still like this when I got back I'd step in.
Contd.
Came back from the wedding a few days later and things were still the same.
So, getting auth to do as I needed to, I stepped into the breach.
I worked 19hr days, scripted the crap outta everything, automated everything.
Contd.
So, getting auth to do as I needed to, I stepped into the breach.
I worked 19hr days, scripted the crap outta everything, automated everything.
Contd.
I had no SIEM, no monitoring, no patch management.
I was working with Win2K3 and XPe, this was pre-Powershell & pre-cloud.
Closest I had to any monitoring was my Sophos Console (SEC).
Contd.
I was working with Win2K3 and XPe, this was pre-Powershell & pre-cloud.
Closest I had to any monitoring was my Sophos Console (SEC).
Contd.
I wrote a system that pushed the Sophos CID out to each store server nightly. To minimise on daytime comms.
I wrote another that managed the roll-out of the patches.
And another to scan a device to see if it was patched.
Contd.
I wrote another that managed the roll-out of the patches.
And another to scan a device to see if it was patched.
Contd.
Wrote another system to re-deploy the Sophos end-point from a share on the local server.
I set the end-point policies to something that was more effective. The down-side of this was that till performance was impacted.
Contd.
I set the end-point policies to something that was more effective. The down-side of this was that till performance was impacted.
Contd.
Management engaged [2-letter UK comms company] for root cause analysis & attribution. Almost instantly they focused on the router, assuming a hack, saying it was likely many months prior.
Knew that was BS. So, let them get on with it, got them out of my hair.
Contd.
Knew that was BS. So, let them get on with it, got them out of my hair.
Contd.
Management had thought that the only way through this was a physical visit to each store, re-image all the devices. This was at the height of their success, there was around 220 stores, with up to 10-12 tills in a store. All in, there was around 2,500 infected end-points.
Contd.
Contd.
I said that would be an option, but, I was confident I could handle this remotely.
Took 2 days to get comms back to a level that enabled 'normality'.
I produced data on number of infections, showing how it was dropping drastically.
Contd.
Took 2 days to get comms back to a level that enabled 'normality'.
I produced data on number of infections, showing how it was dropping drastically.
Contd.
With that many end-points and no real track of how many/where, or even if they were offline & why, it made cleansing tricky. Can't update if it's offline for some reason!
Contd.
Contd.
Two weeks in there were complaints about till performance (due to low-power/resources/WinXPe & end-point policy I had implemented to restrict infection).
Behind my back my boss & his boss went in to the SEC and put the security policies back to how they had been.
Contd.
Behind my back my boss & his boss went in to the SEC and put the security policies back to how they had been.
Contd.
Needless to say, this had a massive negative impact on efforts and started to DDoS stores again.
I lost my shit. Walked out of the office. Went around the block. They knew they had fcuked-up. They didn't challenge me from that point.
Contd.
I lost my shit. Walked out of the office. Went around the block. They knew they had fcuked-up. They didn't challenge me from that point.
Contd.
[2-letter UK comms company]'s forensics team presented their findings - likely an event too far back to tell where/how.
I came out of that meeting and told my boss that was total BS. I said give me 30mins and I'll tell you.
Contd.
I came out of that meeting and told my boss that was total BS. I said give me 30mins and I'll tell you.
Contd.
I used PsExec to pull back all the Sophos end-point logs from the store devices to a single point.
I scanned them looking for Conficker events, looking for "Patient 0"
Cont.d
I scanned them looking for Conficker events, looking for "Patient 0"
Cont.d
When I had "Patient 0" I went through all the Sophos logs. I'd already put in place USB detection/logging, but, hadn't been allowed to enabled blocking.
Through the device logs I was able to see USB events.
Contd.
Through the device logs I was able to see USB events.
Contd.
I traced 'Patient Zero' to the server in a store in the Arndale in Manchester. I had the USB ID of the device.
I could see there was 20mins from event to mass infection.
I was asked how confident I was. 99.9%
Contd.
I could see there was 20mins from event to mass infection.
I was asked how confident I was. 99.9%
Contd.
Someone was dispatched to the store to investigate.
They came back with the USB stick.
I isolated a PC and using Linux (not infectable to Conficker) I was able to get the USB ID.
Perfect match.
Patient Zero confirmed.
Contd.
They came back with the USB stick.
I isolated a PC and using Linux (not infectable to Conficker) I was able to get the USB ID.
Perfect match.
Patient Zero confirmed.
Contd.
Root Cause:
The store had 2x management PC's. One was ours on our network. Other was on the Arndale network. With a very effective "air gap" between them.
The store staff needed the 'footfall' data from the Arndale network.
Contd.
The store had 2x management PC's. One was ours on our network. Other was on the Arndale network. With a very effective "air gap" between them.
The store staff needed the 'footfall' data from the Arndale network.
Contd.
HR had introduced an "Own it, get it done" policy.
So, the store staff had gone to Staples to get a USB stick so they could get the data between the two PCs.
Admirable, really, that they had done this. They weren't to know.
Contd.
So, the store staff had gone to Staples to get a USB stick so they could get the data between the two PCs.
Admirable, really, that they had done this. They weren't to know.
Contd.
Many, many lessons were learnt.
It would eventually be 9mths before I was able to 'call it' on the infection, that there had been no alerts in the previous 2wks.
Contd.
It would eventually be 9mths before I was able to 'call it' on the infection, that there had been no alerts in the previous 2wks.
Contd.
PCI-DSS came along. Suddenly senior management gave a shit about security and [2-letter comms co] sold the business on a managed IDS/IPS solution "you'll be surprised what it finds", they said. It found nothing.
Under my control, that network was clear & tight.
Contd.
Under my control, that network was clear & tight.
Contd.
As store-based ecommerce was on the up a 'store refresh' started which replaced the ageing kit with more capable kit.
Reboot times dropped (and weren't needed as often), end-point policy could remain effective with no noticeable impact.
Contd.
Reboot times dropped (and weren't needed as often), end-point policy could remain effective with no noticeable impact.
Contd.
We had been lucky, in a way, that it was isolated to the stores' network. It never traversed to the rest of the network. That would have scaled the issue up massively.
Contd.
Contd.
I printed out my email warning of this worm and placed it next to my desk. Didn't make me very popular, but, at least it acted as deflection should anyone pull the "Why weren't we warned this could happen?" line.
I have so much more.
But, for now....
END
I have so much more.
But, for now....
END
@threadreaderapp unroll
