How we migrated job processing to a custom service: data layer separation & migration

In recent years, Mews has experienced rapid growth, which has led to complications in managing our software development. As development is now divided among multiple teams, enabling these teams to work independently to avoid bottlenecks has become crucial.

We have taken several steps to support this independence. The first was modularization, which allowed us to split our monolithic codebase into manageable modules. Then, we developed the Atlas platform, which enables fully isolated service development. Today, our product teams can work within our established monolith, build their own module, or opt for full-service isolation through Mews Atlas. Either way, we wanted to break the monolith and decided to prioritize development in isolated services.

While transitioning from a monolith to isolated services brings significant benefits, the process is often complex and time-consuming. One example is our journey transforming one of our key system components, our custom-built job framework (JobFramework), into a fully isolated service.

This article focuses on the final phase of the migration: data layer separation and migration without causing service outages, a phase I personally worked on. In previous stages, my team successfully extracted the domain logic for job scheduling, created API contracts between the monolithic application and the new service, and enabled communication through asynchronous messaging and HTTP requests. However, the service still relied on the monolithic database for data storage, and my role was to address this next critical phase.

Why JobFramework?

JobFramework is a key part of the Mews software ecosystem, designed to manage and execute jobs that run on a one-off, on-demand, or periodic basis. It consists of two basic parts:

• Job scheduler: Its main purpose is to indicate a scheduled task is to be executed at a given time

• Execution logic: A building block isolated in its own library that communicates with the scheduler and ensures the business logic is executed with due care.

Although powerful task management tools exist in the .NET ecosystem, a custom solution was developed to meet Mews’ specific needs. As we grew, however, it became clear that our JobFramework was not optimized for this kind of traffic.

Transition to independent service architecture

We realized that by setting aside the JobFramework scheduling domain as a separate service, we could provide greater scalability and flexibility. This shift would therefore not only improve the way we manage jobs today but also open up new possibilities for integrating job processing into the Atlas platform or other isolated services in the future. We expected that by proceeding with this, we would gain valuable insights and ensure our systems remain efficient as we continue to grow.

To better illustrate what we aimed for in the last phase of the transition, I have outlined the initial and target state we wanted to achieve in the following diagram:

The primary goal was to migrate two critical tables – Job and JobExecution – from the monolithic database to the new service database. This migration had to be done carefully to ensure uninterrupted service availability and maintain data integrity.

Can you manage a seamless migration?

Let’s talk!

Open roles

Initial analysis of data and traffic

Before migrating, we analyzed our environments and found that:

Each environment has about 50-80 million records in the JobExecution table and about 600 thousand to 34 million in the Jobs table. The tables are linked to each other by a foreign key but are not linked to any other tables.

Both tables are very busy in terms of database operations — statistics from the production instance showed that we were making an average of 32 requests per second to the two main API service endpoints. Each of these endpoints performs multiple database operations:

• First endpoint: select from the job table, select from job execution, update the job table, and insert into job execution.
• Second endpoint: select from the job table, select from job execution, and update both the job and job execution.

Migration strategy: CDC vs. dual writing

Considering the target state, the next step was to determine the best approach to migration without impacting the operation of the job scheduler. Given the scale of the migration, the main concern was to minimize downtime and ensure data integrity throughout the process. We needed a strategy that could handle real-time updates and keep the monolithic database and the new service in sync during the migration. Additionally, it was crucial to have a fallback plan in place to safely revert if any issues arose.

After careful evaluation, we concentrated on two potential approaches: Change Data Capture (CDC) and dual writing.

CDC approach

Initially, we considered the CDC function a very appropriate approach. CDC is a natively supported functionality by the MS SQL server we use in the Monolith, so why not take advantage of it?

Upon closer examination, we found that CDC in SQL Server works asynchronously by default, automatically capturing data changes approximately every 20 seconds. Behind the scenes, a scheduled job handles this process by scanning the transaction log at regular intervals and then propagating the changes to designated CDC tables. However, this job cannot be changed in Azure SQL Database, which means that the 20-second interval is fixed and cannot be shortened.

To set up replication, we used Azure Data Factory. It was found that using CDC with Data Factory introduced an additional delay of approximately 10 seconds to the overall replication process, resulting in a replication time of approximately 30 seconds.

Despite the simplicity of activating / setup CDC, there were several concerns we needed to address:

• Table dependencies: For example, if an update occurred in the JobExecution table, but the corresponding record in the Jobs table hadn’t been migrated yet, how would this be handled?
• Performance under load: How would CDC perform under production-level traffic, especially with large, frequently changing tables?
• Replication interval: Could we consistently achieve a 30-second replication interval, regardless of the volume of changes? Is there any way we can speed up replication?
• Integration with Azure Data Factory: How effectively could Azure Data Factory assist with the data migration process?

These issues have highlighted the importance of thorough testing, especially for high-volume tables, to assess the performance of CDC in a real production environment. A key factor was the ability of CDC to process large data sets and high transaction volumes with minimal latency and performance degradation. As questions and concerns continued to arise about this approach, we decided to explore an alternative solution, with the understanding that if it proved ineffective, we could revert to CDC as a fallback option.

Dual writing approach

Dual writing is a technique where an application writes the same data to two different databases or systems simultaneously. Unlike the CDC approach, which passively monitors changes and applies them at scheduled intervals, dual writing actively involves the application of writing data to both systems.

Dual writing introduces several challenges:

• Application logic modifications: It requires adjustments to ensure data is consistently written to both systems.

• Risk of data inconsistencies: If one of the writes fails, inconsistencies can occur. These issues may arise due to failed job scheduling, delays in job execution, or timeouts. Fortunately, our job scheduler was designed with resilience in mind, so occasional failures were not a significant concern.

Despite these challenges, we were confident we could effectively implement dual writing using the ChangeTracking feature in the Entity Framework (EF) kernel. This feature allows you to monitor and manage data synchronization between the two systems with minimal performance impact. Given our familiarity with these tools and the uncertainties associated with the complexity of the CDC infrastructure, we began to increasingly prefer this approach to CDC.

Backfilling and consistency checks

Whatever we decide, there are two common issues to be addressed in both the CDC and Dual Writing approaches: backfilling historical data and ensuring data consistency during migration.

• Backfilling: This is the process of migrating historical data. Both CDC and Dual Writing are, by their nature, reacting to events that are happening in the system in real-time. To migrate historical data, it is necessary to design a service or use a tool that will migrate the data created before CDC or Dual Writing is activated.

• Consistency checks: The goal of the consistency checking process is to compare the data in the source database with the target database and report any discrepancies. The consistency contortion in our case is to give us confidence that the dual-write or CDC is working correctly.

Choosing dual writing for speed and control

After considering both options, we finally decided on dual writing as our main strategy. One of the key factors was speed; many of our jobs ran in seconds, and we needed the data to be available in both databases as quickly as possible. Dual writing allowed us to ensure that any new records created during the migration would be immediately available in both the monolithic and new service databases. This approach therefore minimized the synchronization lag to only a few milliseconds.

Additionally, it gave us a relatively straightforward path to success in terms of implementation. By keeping the existing entity structure and leveraging our familiar repository pattern, we could make the transition without heavily modifying the underlying application architecture.

Using feature flags to keep us safe

We’ve started using feature flags quite a bit in Mews because it’s a great way to experiment with code or test things. In Atlas, we already have a building block for them, built on top of the vendor’s LaunchDarkly solution. For controlling the migration process with the dual-write approach, we only needed two flags:

• Enabling and disabling dual-write: to control when dual-write is being activated/deactivated.

• DB context switching: to switch between database contexts, allowing us to revert to a monolithic database if any problems occur during the migration.

Implementation

The implementation was largely smooth with no major issues. Mews’ backends primarily use .NET technologies, and we aim to keep up to date with the latest versions of .NET and Entity Framework.

As a prerequisite to the migration process, I introduced a new and renamed the existing one to JobSchedulerMonolithDbContext. Both contexts shared the same

configuration to ensure consistency across SQL tables. After generating and deploying the migration, the database setup was complete.

Dual-write mechanism

I used Entity Framework’s built-in change-tracking system for the dual-write implementation, which efficiently tracks entity changes within DbContext. When SaveChanges is called, it generates the necessary SQL queries and executes them in a single transaction.

Nothing was easier than preparing a simple function that takes two DbContexts as input: one as the source (tracking changes) and the other as the target (replicating those changes). I applied this function by overriding the SaveChanges method in both contexts.

I also wrapped this functionality in a feature flag. When the feature flag was off, data was copied from the monolith database to the service database. When it was on, the roles were reversed, with the service database becoming primary and replication flowing in the opposite direction. This allowed us to roll back quickly if any problems arose during the migration.

Backfilling process

For the backfilling process, I integrated it into the scheduler service as a background task. This task was designed to transfer data between the monolith and service databases in batches. Each batch was filtered by a specific time window to ensure only relevant records were processed at any given time. The service used both DbContext instances, similar to the dual-write mechanism, and either created new records or updated existing ones as required. For maximum flexibility, I implemented auxiliary database tables that allowed the backfill process to be paused, reset, or resumed from where it left off. This process can also be run repeatedly, which makes it useful for correcting data inconsistencies and additional synchronization of both databases.

Data consistency check

As a last step, I prepared a data consistency checking service. Again, I worked with both DBContexts; I prepared a generic implementation that receives the entity type as input, columns to be compared on the entity, and another configuration where it was possible to specify paging, time window for comparison, or maximum number of inconsistencies to limit the number of records for a given window. Like back-filling, I exposed an HTTP API endpoint where everything could be configured.

That would be all for the implementation in a nutshell. It could be said that keeping the same structure of database tables during the migration made it much easier. Besides DBContexts and shared configuration, the mentioned FeatureFlags were also helpful. Practically, we did without major modifications to the application logic except for modifying the registration of repositories to the IoC container, since they depend on the specific DBContext type.

The migration process

For a clear overview of the full migration process, I created a BPMN diagram visualizing the steps and decisions involved. This diagram encapsulates the process far more effectively than words alone could. By following this visual guide, you can see how the various elements we’ve discussed come together.

We followed the same migration process across all environments, allowing us to validate and refine the process in less critical environments before executing it in production.

The migration itself took approximately one to two weeks in one environment. Enabling dual writing had little impact on application performance, with write and update delays averaging 100-150 ms due to replication transactions. As a result, the overall APDEX service score decreased from 0.99 to 0.95, which was still within acceptable limits. Turning off the backfill did not affect performance, and we could synchronize approximately 15-20 thousand records per minute, allowing us to complete the historical data transfer in a few days. It is worth noting that we achieved better synchronization performance with smaller batches of around 80 records.

Conclusion

The migration was successfully completed and the service now runs completely independent of the monolithic database. Our implementation of dual writing has proven to be a viable solution, despite the problems encountered along

In the long run, the CDC strategy appears to be a more reliable and less demanding approach to data replication. This is mainly because CDC maintains transactional consistency without requiring significant changes to the application code.

Nevertheless, we continue to explore and evaluate these strategies, as one approach may be more advantageous than the other for different migration scenarios, depending on performance, scalability, and complexity needs.