Migration to the cloud is in full flight. The most complex challenge involved in completing these migrations is not moving the data but migrating the data processing code to work on new infrastructure in the cloud.
The fundamental challenge is to create code that performs the same business process or returns the same result on the new platform, rather than simply making the old code run on the new platform. This traditionally involves a long, manual process of copying the data, converting the code, testing the code and verifying that the migrated code has the same behavior as the original code.
Philosophically there are three different migration approaches – listed in increasing order of risk:
1. Lift and shift
- Move the existing code functionality to the cloud
2. Lift, adjust and shift
- Complete some code redesign during the migration
3. Total redesign
- Rebuild everything from the ground up
Whatever approach you take, keep in mind the Five Hidden Costs in code migration:
Hidden Cost #1 – Underestimating the scope of the challenge
Migration projects are large, and timelines are unpredictable. The decision to migrate must take into account the cost, but without an accurate estimate of the challenge timelines slip and costs balloon:
Questions to ask include:
Where is the data processing code and how much code is there?
- The obvious place to start is in data processing pipelines, but customized code can be found in all manner of places, including BI reporting tools like Tableau.
Does everything need to be migrated? Is some code being run for no good purpose?
- As a result of history and growth, frequently a significant portion of data processing code has been running for no good reason.
How do you structure and plan a large migration?
- Converting the code and testing must be staged. Best in class migrations include a solid, driven roadmap based on end to end data processing. Ideally a DAG that covers all the processing code is available to plan the migration in logical phases.
Hidden Cost #2 – Target Platform Qualification
Can the target platform actually perform everything in a similar way to the source platform? If not then major code rewrites are required, project timelines slip and costs over-run.
Typically the target platform has been tested and qualified based on a selection of data processing pipelines. Inevitably when the entire codebase is explored features not available on the target platform are found. Working around these problems requires the support of subject matter experts (SMEs) on the source platform – access to the wide variety of SMEs become blocking issues which delay the code migration.
With accurate, automated code conversion, an entire code base can be qualified against the target platform without relying on the memory of the SME. Discovering all potential issues early in the process helps scope/cost/solve Amdahl’s law.
Hidden Cost #3: The Non-Standardization of SQL
“Standard SQL” is a myth. It’s tempting to imagine that because a syntax is legal in any two SQL dialects, that it means the same thing and returns the same value. A manual code translation, particularly by an engineer who is not an SME in both dialects, will tend to focus on successful execution first, and rely on a long tail of testing and validation to confirm that the correct calculation is being performed on the new platform. This takes time, and is prone to error.
The hidden cost is not only delays in migrating code, but also issues/errors that are discovered long after the migration is complete. A typical manual migration has errors show up for around a year after switching to the new platform, and it’s common for an automated migration to discover bugs in SQL code left over from a previous migration, or otherwise latent and undiscovered in core infrastructure code.
Hidden Cost #4: Quality and Consistency, or Monday Morning and Friday Afternoon
The cost of ownership of code is not only in authoring and testing it, but also maintaining the code. If quality and consistency is not maintained then there will be ongoing, unexpected maintenance costs on the target platform.
Migration teams must learn accurate, effective and consistent coding patterns on an unfamiliar target platform. With an obvious steep learning curve, the result is often inconsistent code quality, particularly in the first pipelines migrated. This problem is exacerbated at the start of the project resulting in technical debt on what should be a fresh, clean platform. Issues are most frequently encountered around a misunderstanding of the behavior of date-time operations, and changing coding practices as experience is developed.
The result is a long tail of issues discovered in testing towards the end of the migration, which in turn leads to timeline slips and cost overruns, and the worst cases are caused by timeline pressures reducing the testing, leading to bugs remaining in production code which “executes fine,” but does not have the same behavior as the code on the original platform.
Hidden Cost #5: Avoid Stopping the World
Switching over to the new platform is a major milestone that migration project managers strive to achieve. The best outcome that can be hoped for is “things just work.” Of course there is a major risk that problems will arise which stop the core of an enterprise’s data processing.
To avoid stopping the world, the legacy and target platforms can be run in parallel, and the results be confirmed correct. This simplifies testing and increases confidence in the correctness of execution on the new platform. Typically this is completed in stages on segments of the data processing infrastructure. The challenge is to correctly “size the segments” to avoid the project timeline exploding — if the segments are too small then testing cycles take too long. If they are too large then identifying issues takes too long.
If you can address these five hidden costs in code migration then you are on track to a successful migration project.
About the Author:
Shevek is the CTO of Compilerworks