Daniel Zagales, Vice President of Data Engineering, 66degrees, on how to move to a new data platform with minimal business disruption.
Finding reasons to migrate data to a new platform is easy. Data migration can help save money, increase flexibility, enhance reliability and more.
What’s much harder, in many cases, is actually performing data migration. If you’re dealing with data that powers continuous, mission-critical business operations, getting the data from an old platform into a new one with minimal disruption to your organization is no simple feat.
But with the right approach, it’s possible to conquer this challenge.
As I’ll explain, some amount of downtime is unavoidable, but you don’t have to accept days or weeks of disruptions to your business in order to take advantage of a better data platform.
Reasons to migrate data
Understanding what you’re aiming to achieve through data migration is important for adopting an approach that aligns with your goals.
One of the most common objectives for data migration that I see today among companies I work with is saving money. It often happens that businesses have been relying on licensed products to host their data, but after evaluating open source options they’ve decided that a free and open source alternative will allow them to meet their data needs at a lower cost.
You might need to invest a little more effort in management if you opt for an open source solution – but that work is often worth it for businesses looking to streamline data platform costs.
If you’re tied to a specific data platform, you may be wed to a particular vendor’s infrastructure or product ecosystem – this can make it hard to adopt a multicloud strategy.
But if you migrate data to an open source, vendor-agnostic platform, it becomes easier to deploy workloads wherever you need.
A general goal of modernizing architectures and environments can become the motivator for data migration as well. You might want to redeploy your applications using containers, for example, but because you store your data in a legacy platform that was not designed for cloud-native environments, you choose to migrate it so it integrates more seamlessly with your modernized application stack.
Taking the pain out of data migration
Again, determining why to migrate data is often the easy part. Figuring out how to do it is where things get tricky.
So, how do you migrate your data without disrupting your business and depriving yourself of revenue?
The answer is to approach data migration in a coherent, systematic way by following these steps.
- Data assessment
Start by assessing the requirements that your data platform needs to support, then determining which configuration you’ll need to implement on the new platform to meet them. Be sure to consider how requirements may vary as data volume grows and during periods of varying load on your system.
- Data conversion
Data conversion – which means reformatting data to fit the new platform – may be a simple or complicated process, depending on the degree of difference between the old and new platforms. But in most cases, at least some level of conversion is necessary.
You don’t need to take your old data platform offline while performing conversion. To minimize disruption, keep it running but make a copy and convert data based on that. You’ll need to perform some additional conversion later to sync your data just before taking the new platform live, but this approach lets you minimize conversion-related downtime.
- Test deployment
With your assessment complete and (most of) your data converted, you’re ready to run a test deployment of your data on your new platform to validate that it operates as expected and is free of bugs. The test doesn’t have to produce a complete production environment – your goal is to set up a simple test environment where you can push some real-world data into your new platform and assess its behavior.
Containerized deployment of your data platform is helpful here because containers make it easy to deploy a new platform quickly and connect it to your data. That approach is simpler in many cases than standing up a complete production platform.
- Performance comparison
With a test environment up and running, you can run performance tests on both your old and new platforms. This is another step in the validation process and helps ensure that your new platform meets requirements.
- Switching over
Assuming your test environment passes all validations, you are ready to switch your live operations from the old to the new platform. Doing so will require you to take the old platform offline, perform any final data conversions necessary to sync data between the two platforms and, finally, redirect requests from your applications to the new platform.
You should expect some amount of downtime during this process; a few hours is typical. But by carefully planning, preparing and testing the data migration ahead of time, you minimize your risk of unexpected problems and help ensure that you don’t end up with days or weeks of downtime, leading to critical business disruption.
Conclusion: Data migration without disruption
You shouldn’t have to accept downtime for your business to take advantage of new data platforms. With careful planning and testing prior to data migration, as well as data conversion strategies that
minimize the time your data platform is offline, you can achieve benefits like improved ROI or infrastructure modernization without paying the price in the form of data platform downtime.
