Kern County, the third largest county in California, has modernized its data protection for Business Continuity with Veeam.
Kern County is a citizen-centered government that serves and protects nearly a million residents. Protects is the operative word because the county, like most in California, is vulnerable to earthquakes, wildfires and ransomware.
California averages more than 100 earthquakes daily, and while the majority are low in magnitude, many are not, including the earthquake that struck Kern County in 1952. Registering 7.3 in magnitude, it killed 12 people and caused tens of millions of dollars’ worth of damage.
As if earthquakes weren’t enough, more than 62,000 wildfires destroy homes, businesses and property throughout the state each year. Ransomware is another on-going threat. In 2020, ransomware cost US government organizations approximately US$18.88 billion in recovery and downtime.
“We take the threat of disasters very seriously, which means we take data protection very seriously too,” said Mac Avancena, Chief Information Technology Officer for Kern County. “When we moved from a legacy email system called GroupWise to Microsoft Office 365, we worried about data protection because Microsoft doesn’t offer comprehensive backup. Like all governments, we depend on data to serve and protect the public. All 40 of our departments, including fire, sheriff and public health, require continual data access to keep residents safe.”
Transitioning to Microsoft Office 365 was the first step in Kern County’s Digital Transformation. A short while later, the county implemented ReadyKern, a state-of-the-art emergency notification system that alerts residents and businesses to natural disasters and crises. Next the county will transition from a traditional enterprise resource planning (ERP) system to Oracle ERP Cloud.
“In addition to protecting Microsoft Office 365 data, we wanted to modernize and consolidate data protection for every IT system as we moved from an analog world to a digital world,” Avancena said.
“We chose Microsoft 365 because it’s a federally sanctioned system, and we chose Veeam Backup for Microsoft 365 because we felt it’s the best and only choice for data protection. We appreciate that Veeam Software works hand-in-hand with Microsoft to make sure their shared customers have a fully integrated solution that protects the 365 ecosystem.”
Avancena said one of Kern County’s top priorities is Business Continuity, so the IT team deployed Veeam Backup & Replication next.
The Veeam solution
Veeam modernizes, standardizes and centralizes data protection for Business Continuity. Veeam also supports the county’s Digital Transformation and ransomware protection strategy. Additionally, Veeam verifies data recoverability to facilitate compliance with the Health Insurance Portability and Accountability Act (HIPPA), the California Privacy Rights Act (CPRA) and Personally Identifiable Information (PII) law.
“Our goal was to consolidate data protection into one unified environment to protect the integrity of our government, and we achieved that goal with Veeam,” Avancena said. “It’s a good feeling knowing Veeam is our last line of defense in a disaster situation because it’s not a matter of whether a disaster will occur; it’s a matter of when.”
Veeam backs up 2 PB of data across 1,000 virtual machines (VMs) from Hewlett Packard Enterprise (HPE) 3Par StoreServ snapshots to HPE StoreOnce, which provides time-lock data immutability to increase ransomware protection.
“What I find fascinating about ransomware is that everyone wants to get in front of the problem, which is great, but what do you do when you get hit? Are you truly prepared? What’s your Business Continuity strategy?” Avancena asked. “That’s why Veeam is our last line of defense. It preserves our business operations.”
To ensure on-going business operations, Kern County conducts a DR test annually using Veeam DataLabs to create isolated virtual environments for backups. SureBackup tests and verifies backups for recoverability and Staged Restore scans them for malware before they’re restored to the production environment.
“Recovery validation makes it easier to meet regulatory requirements,” Avancena said. “We never purge data, so we like that Veeam is scalable too.”
Avancena said when Kern County deployed Veeam to protect Microsoft 365, no one knew Veeam would eventually protect every IT system.
“Veeam is our standard for data protection,” he continued. “It simplifies and consolidates backup and DR so we have economies of scale, making IT more efficient. Veeam is an important part of a multi-faceted strategy to achieve better benefits for the county.”
We spoke to Mac Avancena, Chief Information Technology Officer for Kern County, to find out more.
Are earthquakes and wildfires the biggest threats to Kern County?
With just our overall geography, natural disasters are part of our on-going business day-to -day process. We have earthquakes and fires.
In that sense, you’re dealing with the perfect storm of trying to maintain Business Continuity and operations in an environment that’s prone to a lot of danger.
The relationship that Kern County has with Veeam is such that they’re our last defense.
At the end of the day, everyone could talk pre-emptively what they’re doing around ransomware attacks and everything else, but when it does happen, or when disaster strikes, you’re going to rely on your trusted tried and true Business Continuity systems called backup.
Do the natural threats you have to deal with every day of the year place an even greater emphasis on data protection?
That makes it significantly more important because when you think about the whole premise and concepts behind Disaster Recovery, you want to have some geographical distance between data centers. And in this case, when you’re dealing with the landmass at 8,200 square miles or so that Kern County encompasses, which for the most part is agricultural and rural, you don’t really have a whole lot of state-of-the-art data center facilities available. So those are all considerations that we have to factor in when it comes to what we’re investing in and how we’re investing in next generation data products.
When you moved from a legacy email system to Microsoft Office 365, why were you worried about data protection?
Data protection wasn’t a concern – data protection didn’t exist. When we moved over to Office 365, this is roughly about four or five years or so ago, what really brought Veeam to the market here at Kern County was they had the product offering that provided backup for all 365.
They were in a lot of regards a trailblazer in the industry for providing data protection in the old 365 ecosystem. That was what started our relationship and throughout the years, with us being customers and them being valuable providers and suppliers for us, they’ve just been able to evolve their products to support the broader Office 365 portfolio and beyond.
Am I right in thinking that in addition to protecting Microsoft Office 365 data, you wanted to protect data for every IT system you have?
We don’t discriminate. We use Veeam as our system of record, if you will, when it comes to data protection and backup. So in addition to our Office 365 footprint which is all cloud enabled, you have our on-prem environments, that are Wintel. So, anything running a Windows-based operating system gets backed up by Veeam and I’m told we’re looking at around two terabytes, give or take, of data that we’re backing up across our system.
We have roughly 1000 VMs. About 10% of them are physical, the rest of them are virtual. And much like what we’re doing with Veeam we’ve actually standardized our overall technology approach with how we’re leveraging different vendors to perform different functions and services.
So simplification and standardization, leads to consolidation, which ultimately gives us a predictable consumption based model that I’m able to charge back to our business. Veeam is all part of that overall strategy to try to simplify how we do business with my organization ITS.
What were the main reasons for you to choose Veeam Backup for Microsoft 365?
It was really a function of who the competitors were out there in the landscape? Because we were in a pickle – we didn’t have a Business Continuity backup solution in place for Office 365. And at that time, the choice was one of two (competitors), Veeam was the incumbent and so we ended up selecting them.
Fast forward to where we are now. It’s really a matter of frictionless types of experiences. So, why would we want to look at a backup solution or Disaster Recovery solution for cloud versus on-prem solutions? I don’t want that. I want a homogenized universal use case for both my on-prem and my cloud base instances.
So those are part of the maturation process around the industry and particularly with Veeam as well. And then in terms of other use cases around what they’re doing, everyone’s talking about cybersecurity and you have to start thinking about immutability of data and making sure it’s not going to be able to get hacked.
Business problems are spawned with what we’re dealing with right now and Veeam has been a tremendous partner, staying ahead of the curve and helping my agency and other customers being able to address and mitigate for those concerns.
Why did you deploy Veeam Backup & Replication to take care of your Business Continuity?
I think it really boils down to a couple of things. It’s the flexibility – that granular level that you’re able to do point-in-time recoveries.
I look at Disaster Recovery from two lenses.
Number one: You look at it from a perspective of incremental changes, there’s going to be use cases where you just need the delta changes for what you lost for wherever that event occurred.
Veeam gives you that flexibility through its integration with vSphere. Everything’s all native – there’s no external APIs as necessary there. And then the secondary use case there is really what happens in a true disaster type of environment if we are totally lights out.
How do we go about rebuilding and re-engineering our systems from ground zero? So there are two very distinct use cases, both of them are critical in my book.
And you know, from my perspective, in terms of our on-going relationship with Veeam, it’s been five years and counting in the hybrid environment that we’re all living in right now. I don’t have to worry about cloud versus on-prem. I just have to worry about making sure that Veeam is solving my business problems for the 40 or so departments that we support and serve.
How does Veeam also support your ransomware protection strategy?
Although Veeam is my standard for data protection, at the end of the day it is my last line of defense.
People will often say you’re only as good as your last backup and so this is no different. I can’t think of a scenario where if I didn’t have the backups for my business-critical systems, whether they were on-prem or cloud, how I would be able to face the board of supervisors, my constituents, my public, and let them know that we were just so irresponsible with the public trust.
That’s the key about government and being in Govtech. The rush around Govtech is that your mission is to serve a population of almost a million residents and as a county organization you’re providing mission critical services for them.
And so we have to start looking at backup not as a chore, but more importantly as a strategic part of our overall planning process for where we invest, how we invest and why we invest. Cybersecurity, the whole concept around hacking and everything else, all that does is accelerate the need for us to make better informed decisions.
And so from my perspective, having this long standing relationship with Veeam, trusting in them that they’re able to use that product to support all of our different businesses across different use cases, just makes it part of my overall strategy around simplification and standardization. We don’t need to reinvent the wheel here – multiple points of entry, multiple points of failure – let’s simplify as much as we can.
Did you achieve your goal of consolidating data protection into one unified environment?
Yes. What’s interesting about government technology in general is that you have different agencies and you have different kinds of governance for technology across each of those agencies.
And so what we tried to do here at Kern from the very get go was recognize that we had some variability in how we approached IT spend, how we approach IT sourcing and how we approach IT standardization.
This is really part of the overall effort, of our playbook, around how we standardize and what we’ve done now is when you look at the 40 or so different departments across Kern County, I’d say about 95-96% of them are already on Veeam. But we have a roadmap to get the other departments on-boarded on to Veeam once their legacy licences expire, and so that’s really part of the appeal.
From an economic point of view, it makes more sense from buying power, from leverage, to be able to come to the table with a use case of 8,500 employees across 1,000 servers all using one homogenized product as opposed to varying types of solutions.
So there’s obviously an economic benefit there and then from an overall administration point of view, I’m not having to reskill and train different staff members for different products. They’re all using the same product so across the board there’s a lot of value in being able to approach things simplistically, which is what we’re doing here at Kern.
