Out Today - Blue Team Diaries: The Big Phish
The Blue Team is charged with defending an organization against an array of technical security threats. The Blue Team Diaries allow the reader to ride along with the Blue Team at Syntatic, a Seattle-based cloud company, who are charged with keeping millions of customer records safe. Based on the author’s real-world experiences, the diaries tell fictionalized versions of responding to actual security incidents. A must-read for anyone interested in computer security or the incident response field.
In this episode, the Blue Team uncovers, via rather unusual means, a large-scale phishing incident that has the potential to become the largest security incident in the company’s history.
Read the opening chapter below, and download the full story on Kindle from your local Amazon store (completely free December 6th — December 10th 2022).
Chapter One
One of the most impactful decisions our team faced on a weekly basis, was the theme of the security awareness email we’d send out to the company every Friday morning. We’d pick a topic, put a few lines of text about it on a colorful template, and then, most importantly, select a meme based loosely on current events, or being in some way related to make the communication ‘fun’. Studies had shown, and by studies I mean us asking people, that this bite-sized approach to security education was popular.
A couple of times we got into trouble because our meme game was a little too close to the bone, or was not ‘compatible’ with the image Syntatic wanted to convey on internal messaging. For example, there was the time we accidentally picked a meme featuring the celebrity CEO of a company we had just landed as a huge customer. There were concerns that the deal could go south if that had leaked out. For the most part, however, the communications were generally well received.
Although the quality of both the message and the comedy contained within suggested a process akin to that of ‘Saturday Night Live’, whereby a week-long intense writing and several rehearsal iterations would precede its sending, the truth was, we’d spend most of Friday morning trying to figure out what to send, before it was too late. This week was no exception. Myself, my self-appointed work spouse Kirsten, and our boss, the CISO, were sitting around sipping on coffees and trying to come up with something relevant to inflict upon the thousands of Syntatic employees, who were just trying to make it to the weekend.
Various themes had been discussed, but none had really ‘landed’ with the group. It was at that point, the CISO had a brain wave.
“I’ve got it! I’ve got it!” He exclaimed. “We need to talk about tax-related security things — it’s tax season.”
He was right, it was late January, and in the United States everyone was gearing up to complete their tax returns. The process of letting the government know how much you think you should’ve paid them over the preceding year, and seeing if they agreed with your conclusion. In terms of annual US employee procedures that people hated, it was up there alongside ‘open enrollment’, the process of selecting your health insurance coverage for the year, or to put it another way, essentially gambling on how sick you expected to be over the coming twelve months and adjusting your out-of-pocket expenses accordingly.
Of course, the tax return process, which involved the sending and receiving of sensitive financial data, often via email, presented criminals with a variety of opportunities for fraud. Our awareness email would be on the topic of being extra vigilant amidst the forthcoming onslaught of scams and attempts at redirecting your potential tax refund.
After much back and forth, we found a scene from the TV show “The Office” that we could use as the basis for the photographic content of the email. A paragraph was quickly written and approved to send by the CISO. After further teamwork in the form of a grammar and spell check, we committed and sent the email to the ‘Everyone’ distribution list. As was tradition, the next five minutes were spent deleting the various out-of-office messages that would come back in after sending such a wide broadcast email.
We all fell silent after sending the email every week, to see if we could hear or see any reactions from the folks around us in the office opening it in real-time. A muted chuckle or a nod in approval in our direction was always welcome but never guaranteed. Today, a sole representative of the engineering team yelled “nice one folks” in our direction, and gave the thumbs up. It was all the recognition we needed. Our job was done for another week.
“Honey Hole?” Asked Kirsten.
“Yep, let’s do it,” I responded.
“I’m out, have a meeting, but you kids have fun,” the CISO added before getting up and gesturing towards a conference room that was steadily filling with senior-level people.
“Ok, just us then, let’s go,” Kirsten said, as we stood up and started to get our jackets on.
Honey Hole was the name of the greatest sandwich place in Seattle. For us, it was a regular Friday treat, and occasionally, a Tuesday and Wednesday treat as well. The only downside, it was on Capitol Hill, which was, as you can probably guess, an actual hill we’d have to ascend in order to get to the tasty sandwiches at the top. In a way, the physically exhausting journey on foot helped us more easily justify stuffing our faces with the giant sandwiches that were to be found once we’d summited. I was also grateful that the walk back to the office after eating was entirely downhill because if things were the other way around, Uber would almost certainly be involved as our official logistics partner.
We ordered our lunch and took a seat. Naturally, we’d have to talk about work things, so that we could justify treating lunch as a work meeting that could be expensed. After a few minutes, the sandwiches and local beers arrived. In addition to being the best-tasting sandwiches in Seattle, they were also the best-named. Both me and Kirsten had selected our favorite, a pulled pork sandwich named the ‘Buford T. Justice’.
Lunch progressed, and we interweaved various work and personal topics as we enjoyed the ‘Justice’, but our fun was to be cut short. I received a text message from the CISO back in the office.
“You both need to get back here,” it read. “ASAP pls”, a follow-up message added.
I turned my phone to Kirsten to show her the contents of the message. We both knew that getting such a communication like that from the boss was extremely rare, and likely to mean one of two things. Either we were in massive trouble for something we’d done, or something serious was going down that warranted our immediate response. Whatever the cause, we both knew that Friday was about to get a lot more interesting, and probably, a lot longer from a working hours perspective as well.
As ever, the coolest under pressure, Kirsten ran to the bar, past the line of people waiting to place their orders, and asked for two to-go boxes for our sandwiches. I managed to squeeze in a couple of extra bites prior to her return to the table, before placing my tragically curtailed sandwich into the to-go box, with the hope I’d be able to resume eating it later.
“Ok, we’re coming,” I responded to the CISO’s text message. “What’s up?”
“Tell you when you get here,” came the reply, which led me to believe that this was something he didn’t want to put in writing.
We barrelled down Capitol Hill as fast as we could, to-go boxes in hand, and feeling a little sick from the expedited departure procedure, which had also included downing about half a pint of beer before leaving. We entered the ground floor of our building and rode the elevator up to our office. I noticed the CISO was still in the same conference room he’d headed into prior to us leaving for lunch, however, the audience of folks in the room with him had thinned out somewhat, which was unusual because it had only been about 45 minutes since we’d left. I assumed the same meeting, with the same attendees, would’ve been ongoing, but clearly, something must’ve happened.
The CISO gestured Kirsten and me into the conference room. Across the table, the Syntatic CTO, and a couple of Vice Presidents from the finance and legal teams were present. I turned my attention to the dual eighty-inch TV screens at the far end of the table, where I spotted two other instantly recognizable faces being beamed into the room via Zoom. The faces belonged to Syntatic’s Chief Financial Officer and Chief Executive Officer. They were sitting next to each other in a conference room in another Syntatic office. They did not look happy.
“What’s going on?” I asked the CISO, sheepishly, not really sure if I wanted to hear the answer to the question.
“Our email this morning about tax scams triggered a few folks in finance to go back and look at some emails from earlier this week,” he explained. “They now think that someone in payroll was phished a couple of days ago.”
My heart sank. Obviously, the folks in the payroll team had a lot of access to very sensitive personal financial information, and if a member of that team had some credentials stolen, that would be sub-optimal. That said, we had mandatory two-factor authentication on all of our remotely accessible systems, so, it shouldn’t have been a death blow.
“How bad is it?” I asked. “What’s been compromised, credentials?”
“We don’t have all the details yet, we’ll need you to dig in, but from the sounds of it, it was something tax related,” the CISO responded.
“Oh. Shit.” I said.
“Indeed,” the CFO interjected.
“Six, can you and the team pull together the details and let us know what you think about this one?” The CISO asked. “I’m having the folks in payroll forward you a copy of the email, and their response to it.”
“Let’s keep the audience as small as possible for the time being,” the CEO added. “Kirsten and Six should be enough to see what’s going on, but I want to be extremely careful around the messaging on this one if it’s serious.”
“Makes sense,” the CISO responded. “Six, we’re going to be in here, can you please let us know what you discover?”
“Sure thing,” I responded.
Myself and Kirsten left the conference room and headed back to our desks. Fortunately, the office had seriously thinned out in terms of the number of people present, as it was a Friday lunchtime after all, so we could talk relatively freely without worrying about anyone overhearing what was going down.
“Alright, let’s see what we got, I guess,” I said, opening up my laptop and checking my email.
Atop the inbox was a freshly forwarded message from Gerard Gibb, whose name I’d seen float by in various emails from the finance team over the past couple of years, but had never interacted with directly. Gerard, it seemed, was our phishing victim.
“Hi there, I was told to forward this to you to look at, I’m really sorry, I thought this was legit,” read the message Gerard had appended to the top of the email thread.
I scrolled down to the beginning of the email chain and beckoned Kirsten in closer to take a look. The first thing I noticed was the thread started on the Tuesday of this week, so four days prior. I read the first email in the chain aloud.
“Hi Gerard, it’s Claire, I need your help collecting some data for an upcoming finance meeting — can you help me out?”
Claire Dawson was the name of the Syntatic CEO, and the person the email claimed to be from. Upon inspection of the sender’s email address, it was clear that this was not from the real Claire. Instead, it was from some random Gmail account, and the sender’s name had been changed to match that of our leader. A classic phishing trick, and one that I really wished Gerard and spotted as quickly as I did. It was too late for that now though.
Gerard had dutifully replied to the fake CEO and asked what data ‘she’ needed. The response came back shortly thereafter.
“I need the W2s for all employees from the previous year, I know you’ve probably just run them tax season, so hopefully that won’t be too difficult for you to put together for me. I appreciate the help!” The scammer had written.
W2s were year-end tax forms sent to all US full-time employees that listed their total earnings for the year, along with the amounts withheld for tax prepayment, and various other financially relevant things. They also had the employee’s social security number on them.
“Surely, at this point, some alarm bells would be ringing, right?” I said to Kirsten.
It looked like Gerard had taken a good hour and a half to respond to this one. Plenty of time to question what he was being asked to do, or to confirm via one of our seventy-two thousand other collaboration tools that we used at Syntatic that the CEO was in fact, the person asking for this information to be emailed to her.
“Hi Claire, I exported the W2s as a spreadsheet, attached. This should have all the information you need, but let me know if you need it in a different format,” came Gerard’s reply.
“Oh shit,” said Kirsten. “He emailed all the W2s, in a single file?! They’d have social security numbers in them, the DLP tool should absolutely catch that and block the emails going out!”
Kirsten was right. We had a data loss prevention tool that inspected outgoing emails to check for sensitive data and hold any such data that should not have been sent out of the company. Social security numbers were of course sensitive, and shouldn’t have been exchanged via email en masse. To that end, we had a rule in our DLP tool that would’ve alerted on and blocked the spreadsheet attachment Gerard appeared to have sent to the scammer. I didn’t recall seeing an alert for a spreadsheet that week, but at the same time, it was entirely possible another member of the team saw it and triaged it.
I opened up my web browser and logged into the web interface of our DLP tool. A couple of searches through the interface revealed no detections or alarms on the day in question. The DLP had missed the outgoing spreadsheet — but how?
I returned to the email chain. The next message in the thread was from the scammer, and it was the last.
“Thanks for your help, this is perfect.”
“So it sounds like they got the file,” Kirsten said. “But how?”
“Great question,” I replied.
How had Gerard managed to unintentionally bypass our typically reliable technical control for preventing these types of disasters from happening? Was the DLP tool misconfigured? Was there some sort of secret exception for the folks in finance that I hadn’t been made aware of?
Something wasn’t adding up. Since I had the entire email thread in my inbox, I was able to download the file attachment that Gerard had sent. I decided that although I’d probably get annoyed by seeing how much other people at the company made compared to me, I should likely inspect it, to see if the social security numbers in the file were redacted or otherwise not fully present.
To my surprise, and delight, some potentially good news. The file prompted me for a password to open it. It was encrypted.
“It’s encrypted Kirsten — that’s why the DLP didn’t get it, it’s encrypted, so it wouldn’t have been able to read it!”
“Well that’s positive because I don’t see any mention of a password in this thread, so presumably the scammer wouldn’t have been able to read it either!”
Having a compromised encrypted file, although still less than ideal, is a far better spot to be in than a compromised cleartext file full of financial data.
I took a deep breath. A potential multi-million dollar and extremely time-consuming bullet may have just been dodged, thanks to a password-protected spreadsheet.
