So nobody can get anything done at the Virginia DMV this week because the system is down. How this differs from any normal week at the DMV, I'll never know. We take a lot of our infrastructure for granted. Power. The Internet. Computers. The postal service (the people with the trucks and the envelopes, not the band). When these systems fail us, it often causes us unexpected inconvenience. During a UPS strike about 10 years ago I was blissfully unaffected, and a little smug about it. However, I was also eagerly waiting for a magazine to come out that had an article about some work I'd done. I kept checking in bookstores but they always had the old issue - a week into the new month. I asked when they'd be getting the new one, and - you guessed it - the UPS strike had bitten me in the ass. FOCUS: When has "the system" being down caused you grief? What was the nature of that grief, and how did you cope/work around the problem? ANTI-FOCUS: When has "the system" being down worked out for you? (Thanks, BL1Y) ALTERNATE FOCUS: Have you ever been the one that brought down the system? Was it on purpose, accidentally, or accidentally on purpose? What were the ramifications?
Focus: When I was working on the Lehman bankruptcy, we would regularly get messages saying that our document system would be going off line for 30-60 minutes. It would, without fail, take about twice as long plus 1-2 hours. While working on this project, we would get chewed out if we didn't bill 8 hours a day, so when the program is down 3 hours in the middle of the afternoon it was a big fucking hassle. If you'll accept an Anti-Focus (the system goes down and it works in your favor): When I was a summer associate at the same firm, the street blew up outside the office (launching pieces of asphalt on to the top of our 48 story building). Most of NYC ignored the story after it was revealed to just be a steam pipe explosion, but a small area, including our office, was in a "frozen zone" with no entry allowed. We got 2 days off of work from the explosion...which happened to have occurred on my birthday.
Sometimes, it's not that the system IS down, but rather that it WILL go down. For instance, let's just say that hypothetically I know "these guys" that are responsible for keeping a huge game's online systems up and running, and yet they have no idea what they're doing. They keep getting more and more information that's being added to those hypothetical systems, and yet they treat that system as if it's a fridge, and just expect it to run without having to do anything to it by way of scheduled maintenance, or silly things like that. Let's just say that, again hypothetically, that a few people with lifetimes of experience have been warning them for 4 months that shit will blow up if they don't do something. We've been assured that these guys, with no prior experience, but armed with the power of Google, are doing just fine and don't need any advice. And, for shits and giggles, let's just say that they're starting to see some strange shit happening, and are scratching their heads in confusion. And now they're starting to ask questions. Those of us that know what we're seeing are now ducking for cover. They are now looking to us to try and help them fix the problems, but they've already waited too long to really do anything about it. And they can't somehow wrap their heads around it. And they're getting pissy over our "not our emergency, so have fun with it" attitudes. They especially don't appreciate the checklist of shit we've given them to watch for as shit implodes. "Told you so" might be petty, but sometimes you just have to beat someone in the face with a 20 lbs sledge hammer to get them to realize that they don't know what the fuck they're talking about. So yeah, knowing something is on it's way down can be sometimes more stressful than it actually being down, especially when you are then looked to to fix it, and do the post-mortem on it. And all of this with a nice long weekend coming up. Joy. Reminds me of something a Russian admin I employed once said... "If you don't schedule maintenance, maintenance schedules you." Not only was he the best admin I've ever worked with, but he was soooo fucking right. His 2nd favourite saying, "9 women can't make a baby in a month", isn't going over too big with those fighting the shit storm either. I find it to be quite humorous though.
The system being down for me is when the computers at work bugger up and leave us without our printers for ringing in orders, printing up bills, and keeping track of various other things. Luckily, I have experience with old-school restaurants that use the ol' handwritten bill system and use calculators to tally up a table's bill. Problem is, in most places I'm the only one, which is a shock because I'm so young. Trick is though, to stay calm. You get panicked, you get fucked. In situations where we're left in the analog age I try and take charge if I'm not in charge normally, assuming the managers aren't prepared or around to handle the issue. If I'm the chef then things go as normal. First step, getting the servers together and prepared to handwrite bills. You go over the steps, making sure they all write bills in a similar fashion, to ensure easier reading. Second step, getting all the handwritten bills for the kitchen in order. You read each bill carefully and make sure each cook has the orders he needs and knows what he's doing. Handwriting is tough to read when you're in a rush, which is why printed bills kick so much ass. So, it's important to slow down a bit and make sure you read each individual item carefully, and that there's a pivot to keep everything in order. Now, while the kitchen is their doing their thing you do a quick tutorial for the servers on how to take credit card payments with the old plastic "sliders", or imprinters. It's surprising how many people have not only never used one, but have never even heard of the fucking things. This includes GMs, who should and need to be prepared for this kind of thing. Next up, get all the calculators in the building together at the various server stations along with a copy of the menu. Most servers don't bother with knowing prices because it's not usually necessary, so having a reference is vital when you go old-school. Final step is to push through and hope things go smooth. Customers are so used to getting taken care of by such mechanical methods that it's a shock when things go a bit slower, which means you're likely to have a couple of pissed off tables. Can't win them all I guess. I don't have any other problems with things "going down", for the most part. I live a surprisingly analog lifestyle, and can handle most problems with ease.
I rarely work from the office anymore so this doesn't bother me anymore but... The area that my company is located in is undergoing ridiculous growth right now. Literally dozens of huge businesses, supermarkets, auto dealerships, and big box stores going in on both sides of the road. Intersections are being widened, new power and phone lines going in... chaos within a 5 mile radius. All of this "work" takes out our phone lines and internet ALL the time. At least twice a week we're offline because of this shit. Of course our old as fuck phone system and slapped together network are precariously balanced normally so whenever shit goes down it takes at least an hour to figure out if it's our stuff or the outside feed. Of course nowadays when you don't get back to customers instantaneously they get worried, pissed, and eventually enraged. Saying, "the system is down" just doesn't seem to cut it either.
Focus: When at my last company (call center) a program was down, someone (I was unaware of this at the time but this guy was a very high ranking officer in a 50,000+ person company) called in and asked when we would be up and running, off the cuff I said I thought it would probably be up and running in the next day or two, pretty reasonable I thought... anyone who works in tech support for a large company probably knows what happened next. The head of IT called in guns blazing demanding my head, inter-department warfare started and my manager had to fight tooth and nail to save my job. One of the senior VP's had to personally come down to our section of the company, read us a script for how we should handle any calls pertaining to systems not working, making sure he drove it home and none of us would fuck up like that again. I learned more about business that week than anyone probably ever learned earning a four year degree. ANTI-FOCUS: Not really due to the system being down, but because of system errors I've had two appreciable arbitrages: 1.) In college I bought $550 dining dollars for one semester, for some reason they gave me $1,100. Hello sushi every day! I probably have more mercury in me than a rectal thermometer. 2.) Early into my career they accidentally overcharged me for medical insurance. When they corrected the error someone accidentally left out the decimal place and I got about $1,000 instead of $10 back. Hello big screen TV! In retrospect that was pretty shitty of me and I should have let them know about the error. Of course it's easy to say this now that I actually make money instead of being dead broke like I was then.
When I moved from Newfoundland to Ontario, I had to have my driver's licence switched. So... After putting it off for months, I finally chose a day to do so. I was only going to miss an hour or so of work, so it wasn't a big deal. I went downtown, I took a number and the time flew by. I filled out the paperwork, took the eye exam and that should have been that. I was going to be back at the office in no time. The last thing left was for the woman to contact the Newfoundland Department of Transportation and Works or something. The change confirmation or whatnot had to be sent to them electronically. It just seemed like their server was down. She asked me to take a seat and wait and she'd try again. And I did take a seat. For 4 goddamn hours. I couldn't leave until the confirmation went through. I had to wait or I had to come back another day and do it all over again. I was already there, so I sat there like an idiot for 4 hours. I kept checking with her, too. "Looks like their system is still down. It won't go through. I'm sure it'll just be another few minutes." I was ecstatic when she finally waved me over. So ecstatic that I went straight home and didn't even think about going to work. My day was shot, anyway. I worked in an outbound call centre when I lived in New Brunswick. One night, I was working and I noticed that it had been about 5 minutes since I'd taken a call. Strange. Then it was 10 minutes. I called my supervisor over and he checked his computer. I was still logged into the queue and there didn't seem to be a problem. He just told me to sit tight, so I did. I sat there and got paid to not take a single call for about 2 hours. I kept bringing it to his attention and he didn't care. When he finally looked into it further, he didn't have an answer for me. He said anything that could have possibly happened to bump me out of the queue didn't. It was just some odd glitch that he'd never seen before. That was a fun night.
Manufacturers always try to cut their shipping costs. My largest customer was just bought out, with the new CEO rebidding a couple thousand shipping lanes. Of course they gave everything to the lowballers, pricing out all their current vendors who've taken care of them for years, some decades. Right now, they'll be fine. But in a month or two, when capacity for equipment tightens up, they'll be fucked and I'll hear from them. "I told you so" would be so sweet to say, but I think an increase in pricing will do the trick.
FOCUS: When has "the system" being down caused you grief? What was the nature of that grief, and how did you cope/work around the problem? Back when I was a call center insurance agent, we had various systems that went down numerous times every month. Between the DMV going down and not being able to electronically pull driving records, to our main record keeping system going down (which we used to launch all other programs from...good going douche bags, now we can't get anything done because we can't open the programs any other way) and being forced to take down the callers information (address, vehicles, prior insurance, social security number, coverage) and then inform them we'd call them back with a quote once our systems were finished "updating", because company protocol dictated we do so. ANTI-FOCUS: When has "the system" being down worked out for you? When I left the call center gig, I was paid for an extra 2 weeks of work. They didn't catch the error until 6 months later. I ignored them and they never listed it on my credit report. It serves the bastards right for firing me because I went to my grandmother's funeral instead of coming to work, like I informed them I would.
My career as an adult has been entirely about managing 'the system'. I was heavily involved with an Olypic games, a dozen major international sporting events, 100,000+ websites, several million email users and websites serving content to 500 million unique visitors over the course of a 24 hour period. I broke an olympic games webcast. I broke a national semi-final webcast. I took out 80% of the server farm handling the websites and email above once. My boss had assembled a PC from spare parts, and asked me to image it for use. I was new to the company, so I shrugged and said sure. Assumed he knew what he was doing, rested one hand on the front of the bare metal case, and hit the on button. The idiot had put the power button in upside down, so the active element was hard up against the frame. For those of you who know how electrical things work - you know what happened next - I flew across the room and hit a wall, went to hospital when I woke up and spent the next several months calling my boss 'cunt' (our overboss agreed that it was a fair call, all things considered). The less obvious part of the story was that the PC was plugged into a build bend power point that was on the same phaze as (and improperly isolated from) the server farm - our sparky found himself replaced fairly shortly after that. It took about 6 hours to get those 80,000 affected websites back online.
Anti-Focus: In college, I would never buy ink for my computer. This meant that if I needed to print out a paper, I went to the school library and used theirs. Once, I completely forgot about a 5 pages paper I was supposed to write about how graphic novels have become a form a literature. Thankfully, the school's computer system was down, and I got an extra week to write the paper.
Focus: Back in High School I had worked my butt off to make the varsity tennis team. 4 years of playing shitty tennis culminated in me throwing all my effort into making the varsity team. Yup. I was that D-Bag on the court grunting, throwing myself around and trying harder then I ever thought possible. All to get the lowest spot on the team. Anyway, going into the weekend after my amazing "Miracle on Ice" like victory I casually read in the news that all the doctors in the Private Practices would be going on strike to get more money from Insurance Companies. Cue the Gods of Irony reaching down from the heavens and kicking me in the nuts, and I start to get sick on sunday. Thinking it was only just a cold or possibly a mild flu, I ride out monday at home thinking it would be better for Tuesday. Come thursday and I am as sick as I had ever been in my life (at that point). One trip to the hospital later I am hooked up to all sorts of machines and IV's. I had Strep Throat and am the closest to death anyone in all of the 1st world countries had been in at least 100 years from this disease. All of which would have been solved had my private practice doctor been there on monday and spent the 10 seconds it takes to look in my throat for Strep Throat.