Human Error and the Google Cloud $135 Billion Account Deletion: How to Avoid Costly Downtime and Reputational Damage

There’s an updated page on the $135 Billion Australian pension fund UniSuper’s website that no one ever wants to publish. 

The page titled “UniSuper services fully restored” links to a joint statement from UniSuper CEO Peter Chun and Google Cloud CEO Thomas Kurian, dated May 8, 2024. The statement says, “UniSuper and Google Cloud understand the disruption to services experienced by members has been extremely frustrating and disappointing.” 

But the damage was done, causing two weeks of downtime and unimaginable damage to UniSuper’s reputation. This incident highlights the importance of unified data protection because it wasn’t caused by ransomware or cyberattacks. It was caused by “an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription.”

Human Error: The Root Cause of Most Outages

In other words, human error. The statement continues, “This is an isolated, ‘one-of-a-kind occurrence’ that has never before occurred with any of Google Cloud’s clients globally. This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again.”

But human error-caused incidents will continue. The Uptime Institute’s 2023 Annual Outage Analysis says: “Uptime estimates (based on 25 years of data) that human error plays a role in two-thirds to four-fifths of all outages.”But if you can’t prevent human error, how can you mitigate its effects? This is where the UniSuper story illuminates the problem—and the solution.

Data Protection and Downtime Prevention Depends on Redundancy

The UniSuper statement also noted that “UniSuper had duplication in two geographies as a protection against outages and loss. However, when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies.” 

The statement notes that “UniSuper had backups in place with an additional service provider. These backups have minimized data loss and significantly improved the ability of UniSuper and Google Cloud to complete the restoration.” However, for UniSuper, recovery required “an extensive recovery of our Private Cloud, which includes hundreds of virtual machines, databases, and applications.” 

Unified Data Protection and Redundancy: Multilayered Defense Against Downtime

The nearly two weeks that UniSuper’s private cloud was down must have been excruciating. While there is no way to ensure that a human error or cyberattack won’t cause downtime at your organization, you can take several steps to minimize the potential for outages and data loss. 

That starts by embracing the 3-2-1-1 backup strategy—of which Arcserve is a major proponent. This strategy adds layers of redundancy to your data protection efforts. Here’s how:

• As UniSuper did, keep at least 3 copies of your data (one original and two copies)
• Store your backups on 2 different types of media (network-attached storage, tape, or a local drive, for example)
• Keep 1 copy offsite in the cloud or secure storage
• Ensure 1 copy of your data is kept in immutable storage

Immutable storage matters most because it gives you a last line of defense against data loss and downtime.Your data is saved in a write-once-read-many (WORM) format that can’t be altered or deleted. So you can be confident it will be available when all other attempts at recovery fail.

Every Cloud Data Protection You Need

The UniSuper private cloud “includes hundreds of virtual machines, databases and applications.” Arcserve Unified Data Protection (UDP) is built to protect against data loss and extended downtime across cloud, local, virtual, hyperconverged, and SaaS-based workloads.

The software makes it easy to scale hybrid business continuity topologies, including service and cloud providers. You create data stores on the recovery point server (RPS), add the nodes you want to protect, a storage destination (including one immutable copy), and a plan. 

Most importantly, Arcserve UDP protects your data across many cloud platforms while offering Arcserve Virtual Standby. This feature converts your backup data into a virtual machine format and creates a virtual machine that can take over when the source node is down. It also prepares a snapshot of the recovery points to make data recovery easier. 

Get Expert Help

Arcserve Technology Partners are data protection experts who can help you ensure your data is safeguarded and recovery is fast, efficient, and effective.

Find an Arcserve Technology Partner.

To learn more about Arcserve UDP, request a demo.