Read the first chapter of Blue Team Diaries: Shellshocked

Enjoy the opening chapter of Blue Team Diaries: Shellshocked. Learn more about series, and how to read on, here: https://mybook.to/shellshocked

Shellshocked: Chapter One

I’d been arriving at the park and ride progressively earlier each day for the past month. It was currently a slither past six o’clock in the morning, and I was able to slip into one of four remaining parking spaces. This was getting ridiculous. They told us we shouldn’t be driving into Seattle, so everyone was taking the bus. But good luck finding a way to get on the bus in the first place, unless you wake up before dawn, or simply decide not to sleep anymore. I parked up and stepped out of the car. The cars immediately behind me filling up the remaining spaces. I caught a glimpse of the unlucky commuter who realized that they wouldn’t be parking and riding that day. Well, not parking here, anyway.

Headphones firmly on, podcast engaged. After a short wait, I was on the bus, standing room only of course. We were heading into the city. The standard weekday routine had begun. The next big milestone for me would be joining the line at the Starbucks on the bottom floor of my office building to get that sweet sweet first sip of caramel macchiato.

I glanced up to check on our progress, but I could barely see out of the windows. It was raining, for a change, and that meant that we were very much steamed up. I should’ve known better than to rely on the real world for positional information. I opened up Google Maps on my phone to confirm my location, I was about 25 minutes out, with no more stops.

Suddenly, a grinding noise came from the back of the bus, loud enough to hear over the podcast, followed by an unwelcome sensation that we were slowing down. We were in the HOV lane, which was on the far left-hand side of the freeway. Not an ideal place to spend your morning, or afternoon, or evening, for that matter. The slowing bus came to a complete stop. I was able to stand on my tippy toes and look out the front window. There was no traffic ahead of us that would’ve caused us to make this unscheduled pause, so it was obvious something was up.

The bus sputtered a couple of times as the driver attempted to restart the engine. I’m no bus mechanic, but I could tell it sounded sick. Things were getting serious, so I took off my headphones. The person who I’d been sharing a grippy bus pole with noted that I would now be able to hear them, and took full advantage of the opportunity.

“That doesn’t sound good,” he said.

I had not expected to need to talk to anyone this early, so I needed a moment to reconfigure my brain into the right mode and respond.

“You’re right, it doesn’t,” I replied, insightfully.

The traffic was speeding past us, and now and again a car that had been in the HOV lane behind us would pull out and scoot past when they could. At this point, it was very clear that we were a rather large obstacle. Before I was forced to make further small talk with my fellow commuter, the driver made an announcement.

“Er, yes, well, as you can tell something is not right with the bus, I’m talking to base now, and they’ll send assistance, so hold tight, and we’ll have this figured out as soon as possible, thank you.”

A collective groan reverberated around the inside of the bus. I had two thoughts; best case scenario, I’d be late to the office. Worst case scenario, I’d be dead, because someone would come flying down I-5 and pile into the back of us. I opened up Slack on my phone, and pinged the team, letting them know I was currently stuck on the freeway and didn’t expect to be moving any time soon. Not that it mattered, because ‘late’ to me, is a more typical start to the workday for others.

The bus driver had attempted to restart the engine a few more times but was still struggling. I got a Twitter notification from the Washington State Department of Transportation’s traffic news account.

“A stalled bus is currently blocking the HOV lane on I-5 just south of Mountlake Terrace,” it read.

“Yes, I am aware,” I mumbled to myself.

The tweet was accompanied by a photo from a nearby traffic camera, which meant I was now staring at a picture of the back of the very bus I was stuck on. What a time to be alive.

The bus driver gave another PA update, in which he mentioned that a second bus was on its way for us. This meant we’d have to transfer to said bus, on the side of an extremely busy freeway, in the pouring rain. I almost wished it was me that had missed out on a space at the park and ride this morning.

I continued to do what I did best while on the bus and craned my neck down to scroll through my phone. I was doing my usual round of checking news sites and social media when a clump of tweets appeared that caught my eye.

“CVE 2014–6271, patch now!! Patch everything!!,” read one.

“Oh my, this Bash bug is going to be significant — he’s a POC for getting a reverse shell via CGI,” read another.

It was not unusual to see chatter specific to newly discovered vulnerabilities on my Twitter feed, after all, one of the reasons I followed some of the folks I did was to get that information specifically. What was unusual, however, was the rate at which the chatter about this particular vulnerability was flooding onto my screen. Unusual, but not unheard of. About four months earlier, another vulnerability, Heartbleed, which turned out to be a very big deal had manifested itself into my daily routine in very much the same way. That said, I wasn’t stuck on a fucking bus at the time. Was this another Heartbleed? Was this going to be another scramble to patch pretty much everything we had exposed to the Internet? I was worried it would be, and I wasn’t looking forward to having that conversation, especially with Heartbleed only having happened a few months earlier. It was still very fresh in the minds of a lot of our technical teams that I’d disrupted them for a couple of days by ‘pulling the alarm cord’.

I needed to learn more, but I was well aware that I was somewhat constrained by my current surroundings. I put my headphones back on, to block out some of the idle gossip that was occurring between my fellow bus prisoners. I opened up an article on the newly disclosed vulnerability that permitted remote code execution in Bash, a nightmare scenario.

The articles explained how the vulnerability impacted all versions of the Bash shell, the most commonly used Unix login shell, from version 1.03 to the current version. Further reading revealed that this meant the bug had been around for about 25 years. This vulnerability was almost as old as I was.

Bash was everywhere. This was going to be a big deal, especially if it really was as bad as everyone was saying. Syntatic was not going to be immune from having to respond, and a big, well-coordinated response would be needed. If there was already proof of concept code out there for this vulnerability, it wasn’t going to be long before someone started to exploit the vulnerability at scale. We’d seen this with Heartbleed, so it didn’t take much of an imagination to see it happening again with this new Bash bug.

I started to compose yet another Slack message in our team channel, I needed to get as many eyes on this vulnerability as possible, and preferably, eyes that were attached to someone who was sat in front of an actual computer, not crammed into a bus relying on a smartphone.

“@here,” I wrote before erasing and upgrading the urgency of the Slack message to “@channel”. “It’s Heartbleed all over again, but possibly even worse, this is going to be big and we need to respond to this. The first priority is going to be anything externally accessible, then we’ll look at the other stuff. Oh, and I’m still stuck on the bus.”

In addition to the Slack message, I sent a text message to the two folks higher up the chain. My direct boss, the CISO, and their boss, the CTO. I told them to check the Slack messages and be prepared for a busy day of patching.

I started to prepare a second message in Slack. This one was to be a broadcast message to a wider audience at the company, the idea was to prepare as many people in technical leadership positions to be prepared to send me engineers to assist in a significant patching effort. As I started to type, I became distracted by another bus that pulled in front of our stranded one. It was a positive sign, but we’d still have to get on that bus. That would involve taking a rush-hour stroll on the interstate the length of the replacement bus. I considered it unlikely the transit agency would allow 100-ish civilians, most of whom were adorned with dark-colored North Face or Patagonia jackets to take that walk without some sort of assistance. I was correct.

“Folks, the bus is here, as you can see, but we need to wait for a State Trooper to come to close down the lane you need to walk down to get to it, they are on their way,” said the bus driver, over the PA.

The Washington State Patrol had a habit of being there when you didn’t want them to be, but taking their sweet time when you needed them badly. I returned my focus to my drafted Slack message, and continued to type out my Bash bug “SOS”. By now, a couple of my team members had appeared online and had started to relay further information about the severity of this newly discovered nightmare.

“Holy shit, this really is an 11 out of 10,” one of my engineers had commented.

“What we need to do is start the process of notifying service owners so we can get an ETA on patches. Use the service list template on our Confluence page, it should be pretty up to date. Also, let’s notify the networking team, and also IT because we’ll probably have some business systems that’ll need a patch as well,” I wrote.

One of the positives to have come out of the Heartbleed response, we now had a fairly accurate list of all our external facing systems, and who within the company was responsible for them. We thought we’d had that before Heartbleed, but as it turned out, we had quite a few blind spots.

“Can someone please have the operations team start a P1 bridge as well?” I asked. A P1 phone bridge was a signal to the company that we were not messing around. Automatically, all on-call engineers would be notified, and they’d be expected to join the call for instructions. I generally avoided starting P1’s when possible, but I knew I needed one now. They were noisy, costly affairs, that the person who triggered them would have to justify in a later post mortem, and experience that was, in my opinion, designed to put people off. We actually used the expression ‘stealth P1’s’, because a lot of times folks would run a P1 response without actually going through the process, simply because they didn’t want to deal with the fallout. All perfectly normal corporate culture stuff, I’m sure you’ll agree.

I was growing increasingly frustrated. My laptop, the most crucial tool in my incident response arsenal, was in my backpack, and completely unusable in my current state. A P1 bridge call was about to start that I’d asked for, which meant I needed to be on it, but I was stuck on the bus. I was about to become everything I ever hated, someone who would have a very loud work call while on public transportation. It was for the greater good, I kept telling myself.

I hadn’t noticed, but a State Patrol officer had pulled alongside us, stopping traffic in the lane closest to our stricken bus.

“Ok folks, let’s get you off this bus and on your way, be careful please,” the bus driver yelled, beckoning us to the front door which had been opened. This was the Pacific Northwest, so of course, everyone was being extra polite and letting everyone else go in front of them, further slowing the evacuation. I’d hate to be on a burning aircraft that had just crash-landed at SeaTac, no one would want to get off first for fear of appearing rude.

It was finally my turn, I shuffled out of the bus and glanced around to make sure I wasn’t about to be hit by something. It was cold and wet, but I was making progress on my commute for the first time in a long time, so I was relatively happy. Traffic was moving slowly since most people were rubbernecking our broken-down bus, surrounded by the State Patrol and a transit supervisors support vehicle.

I got onto the shiny new bus, which had been empty so had clear, non-steamed up windows. Suddenly, a rare sight. In the shuffle, since I’d been standing in the aisle on the last bus, I’d be rewarded with a seat on this bus. I made a beeline for it. Typically, I’d put up with standing, but there was very clearly an Internet crisis happening, and if I was going to be constrained to my phone for longer than anticipated, at least I’d be able to use both hands now.

I checked my email. As expected the P1 bridge call had been started, and I’d been given the all-important link to join it, which I did. I was met with the sound of idle chatter between a couple of engineers and the operations team.

“Six wanted this bridge, hopefully, they’ll be here in a second to explain why,” said the operator.

“Hi this is Six,” I said. “Thanks, everyone for jumping on, especially those of you who have been woken up early. I apologize for any background noise, I have been stuck on a bus on I-5 for some time, and am still on my way in. If you all think back a few months to the last mass patching event we did for Heartbleed, well, guess what? We’re doing it all over again.”

There was a very clear audible sigh from someone on the end of the phone. Throughout the call, I’d been hearing beeps as more and more people connected to the phone bridge.

“What are we patching this time, and do we know if it impacts us for sure?” Asked a voice on the bridge.

“Bash,” I replied. “We’re patching Bash. Test the patch in staging first and then do production please.”

“What is that, I’ve never heard of it? Do we use it?”

“Bash as in the Bash shell? Asked another person.

“Yes, the Bash shell,” I explained. “So, yes, we do use it, and yes, so does the entire Internet. That’s probably the saving grace at the moment. There are so many targets out there to choose from, we might slip under the radar for a little bit, but not long.”

“Six, when this happened with Heartbleed, you said that would be a once-in-a-lifetime event!”

“Well, I regret to inform you, you have died and come back in the same job, appearance, and life as before,” I replied sarcastically. “Now yes, I know this is frustrating so soon after Heartbleed, but just like I said then, we need to tackle this well, and we need to tackle this early. Please update everything you have to include the fixed versions of Bash that have been released today. My team can help you determine if your system is vulnerable, but just assume it is for now.”

The last of the rescued passengers had entered the bus, and it was time to go. The engine on this bus had been running since we got on, so there was no question as to its ability to fire up. The doors closed, and the bus pulled away. Just like the Syntatic response to this vulnerability, we were underway, but there was still some distance left to go before I’d feel anything close to comfortable.