Things happen - react well both in the moment and afterwards.
Give me a RED CROSS!
Amazon AWS experienced a service disruption to CloudFront.
Things break, it happens. Given the size and prevalence of AWS the whole internet knows it.
Nothing can go wrong quietly in a corner for them.
If you are reading this it is all good again ;)
“It's not what happens to you, but how you react to it that matters.”
I'm not worried about how they will respond to the technical issues involved - they
have an amazing track record there.
However while it was happening did it seem acknowledged ?
Millions of dollars of productivity and commerce no doubt disappeared today.
Untold thousands of system administrators and network engineers checked DNS, pinged
sites and reached out to colleagues and friends in the industry.
On Twitter, IRC rooms and Slack channels people acknowledged and agreed something was happening.
"works for me" - "not for me" - "I can't get here" - "can't get there!"
while an hour into the problems the provider showed this:
Everything is Green.
Everything is OK. Hang on what is the subtle Information mark?
what is that?
On twitter (@AWSCloud)[https://twitter.com/awscloud] sat silent.
What is going on guys ?
Sure its 5pm but is everyone in cars driving home ?
As a customer we just want to know is it me or is it you. Can I do anything?
Does the corporate communication plan for outages explicitly forbid public acknowledgement ? Possibly.
In the morning news and over the next few days we'll hear about what was down and
what it impacted. Big numbers will probably get thrown around and then the next news story will hit.
In boardrooms executives will grill technical staff about the resiliency of their
solutions and what did they know when???? and what will they do about it ?
Competitors will pounce and attempt to entice and erode customers away.
The response after the fact I'm sure will be great.
Systems will improve. Quality will enhance. I just hope one of those pieces is how they communicate
during these "should never happen" scenarios.
It's not what happened as much as how you respond when you do.