Moving Data Between Systems That Keep Changing

Our last engagement for Data Migration Services took fourteen months. During that time the source system had two major upgrades, the target was reconfigured twice, and a third system we integrate with was replaced entirely.

Data Migration Services assume a stable world and never get one. Here is how we plan for that now.

Assume the source will move

It is still in use, which means it is still changing.

Fields get added. Validation rules change. A supplier upgrade alters how a value is stored. Each of those invalidates part of your mapping, quietly, unless somebody is watching.

We now profile the source monthly throughout the programme, not once during discovery, and compare against the previous profile. Three changes were caught that way, each of which would have produced wrong data at cutover.

Freeze what you can, and know what you cannot

A negotiation worth having early.

Some changes to the source can be deferred until after migration. Regulatory ones cannot. Commercial commitments cannot.

We agree a change freeze covering everything discretionary, with a named person who approves exceptions. The exceptions still happen and they are visible rather than discovered during reconciliation.

Version the mapping like code

The discipline that makes change manageable.

Our mappings sit in version control, reviewed, with every change recorded and reasoned. When a source field changes, the mapping change is a reviewed commit rather than an edit somebody made in a spreadsheet.

That record also answers the question that always arrives later. Why does this field contain what it contains?

Rehearse often enough to notice

Not once, and not at the end.

We rehearse monthly from the midpoint, at full volume. Each rehearsal reconciles and reports, which means a change in the source shows up as a reconciliation break within weeks rather than at cutover.

Programmes that rehearse twice discover their source has moved on the weekend it matters.

Protect the people who know the source

The scarcest resource in any migration.

Usually two or three people understand the source system properly, and they are also needed for its day to day operation.

We name them in the plan, agree their availability in writing with their manager, and record what they tell us rather than relying on asking again. One of ours left in month nine, which would have been a crisis rather than an inconvenience without that record.

Data Migration Services engagements that assume unlimited access to those people are planning on something nobody has agreed.

Keep the target configuration under control too

A failure we caused ourselves.

Our target system was being configured by a different team while we built mappings against it. A field was made mandatory, and our migration failed on a quarter of the records.

Target configuration is now frozen alongside the source, with the same exception process. Both teams work from one change log.

Plan for the surrounding systems

The ones nobody includes in scope.

Our migration also affected a reporting platform, two integrations, and a client application that read directly from the source database.

That last one was found in month seven. Any Data Migration Services engagement should inventory what reads from the source, not just what the source contains, because direct database readers are invisible until they break.

Separate the architectural change

A boundary we hold firmly.

Our estate was simultaneously being restructured, following the familiar guidance on moving from monolith to microservices, and the two programmes were nearly merged for efficiency.

We kept them apart. Changing where data lives and changing how applications are structured at the same time means that when something is wrong, nobody can say which programme caused it.

The architectural work waited six months and both went better for it.

Watch what the data will be needed for

A requirement that arrives mid programme more often than not.

Partway through ours, the organisation decided it wanted the migrated data available for analysis and eventually for AI use. That changed what we documented rather than what we moved.

Field definitions, completeness measures and permission positions became deliverables. A later Healthcare Generative AI proposal elsewhere in the group was approvable only because an equivalent migration had produced exactly that record.

Expect the business to change too

Not only the systems.

During our fourteen months, two product lines were discontinued, a pricing model changed, and an acquisition added a customer category nobody had mapped.

Scope decisions therefore need revisiting rather than being settled once. Ours are reviewed quarterly with the business owners who made them.

What we plan for now

●        Monthly source profiling throughout, not once.

●        An agreed change freeze with a named exception approver.

●        Mappings in version control, with reasons recorded.

●        Monthly full volume rehearsals from the midpoint.

●        Target configuration frozen alongside the source.

●        An inventory of everything reading from the source.

●        Architectural change kept in a separate programme.

Common questions

Does the sector change the approach?

It changes the governance. Regulated downstream uses require documentation that a commercial migration often skips.

How long is too long for a migration?

Beyond about eighteen months, the world changes faster than you can migrate. Break it into phases instead.

Can a change freeze really hold?

Not completely. It holds for discretionary change, which is most of it, and makes the rest visible.

How much source access is needed?

More than anybody budgets. Data Migration Services depend on two or three people who also run the source system day to day.

Who should own the change log?

One person across both systems. Two separate logs produce two partial pictures.

Does downstream use change the plan?

It changes the documentation rather than the migration. Healthcare Generative AI and similar uses need field definitions and completeness recorded, which a migration produces cheaply and usually discards.

What surprises programmes most?

Direct database readers. Data Migration Services scope the data and frequently miss what else is reading it.

What fourteen months taught us

The mapping work was reliable. What made the programme difficult was that the world it was mapping kept moving while we worked.

Monthly profiling, monthly rehearsals, a shared change log, and a separate home for the architectural work are what turned that from a recurring surprise into something we could plan around.