Cross-service data migration (adjustment reasons)

Hello everyone,

Yesterday we had a discussion about the best approach for the adjustment reason migration from ref-data to stock: https://openlmis.atlassian.net/browse/OLMIS-2830

There are 3 parts to this migration:

  1. Schema changes in requisition

  2. Transfer of reasons from ref-data to stock

  3. Migration in requisition, that switches the UUIDs so that the point to stock reasons, instead of ref-data reasons

The first one is obviously a Flyway migration that run in requisition. The other two are more tricky since they deal with cross-service data migrations - and we want to avoid services operating on each others databases.

There are two ways to do steps 2 & 3:

  1. Migrations in Stock and Requisitions respectively. These would have to be Java based migrations (Flyway supports that) that fetch the data from the other service using the API and then doing the inserts. These would have to wait until the other service starts up - they would have to hold up the start of the service they will run in. This can be a bit complex, since we haven’t done these in OpenLMIS yet (spring context access might be limited) and there are cases we want to account for, for example we don’t want them to run when running just the single service or in tests - a solution is either use a flag or base that on consul checks, but obviously this can get increasingly complex. These migrations however would be transparent to the implementer.

  2. The easiest option probably - a standalone script that does the data transfer. Ideally the implementer would just have to populate the config file with something like database credentials for the service and then just boot this script up with one command - it would do the necessary data transfers between service databases. If it helps we can provide this script as a docker container. The downside is obviously that the implementer will have do the migration manually. In this case it will probably only be Malawi, but I suppose cross-service migrations are a topic that can come up in the future again.

I spoke briefly with Josh about this on Slack, and we kinda levitate towards the script solution because of simplicity. What does everyone else think?

Regards,

Paweł

···

Paweł Gesek

    Technical Project Manager

     pgesek@soldevelo.com / +48 690 020 875

Hi All,

At today’s Tech Committee we discussed this. Paweł confirmed with Team Malawi (Sebastian) that Option 2, Stand-Alone Script for Data Transfer, would work for them. The Tech Committee attendees agreed that Option 2 is appropriate. We listed a few specific areas that need to be documented or addressed:

  • release notes? (need to provide documentation of this migration script to include in release notes)

  • how to run? (document how to run it)

  • should it support multiple data sources? (Malawi’s only 1, is there ever any reason to have this work one more? nobody at Tech Committee suggested any more)

  • is their any sort of framework or container that’s re-usuable? (TBD)

I will updated the ticket accordingly:

https://openlmis.atlassian.net/browse/OLMIS-2830

…and we will proceed.

-Brandon

···

On Friday, August 4, 2017 at 5:07:31 AM UTC-7, Paweł Gesek wrote:

Hello everyone,

Yesterday we had a discussion about the best approach for the adjustment reason migration from ref-data to stock: https://openlmis.atlassian.net/browse/OLMIS-2830

There are 3 parts to this migration:

  1. Schema changes in requisition
  1. Transfer of reasons from ref-data to stock
  1. Migration in requisition, that switches the UUIDs so that the point to stock reasons, instead of ref-data reasons

The first one is obviously a Flyway migration that run in requisition. The other two are more tricky since they deal with cross-service data migrations - and we want to avoid services operating on each others databases.

There are two ways to do steps 2 & 3:

  1. Migrations in Stock and Requisitions respectively. These would have to be Java based migrations (Flyway supports that) that fetch the data from the other service using the API and then doing the inserts. These would have to wait until the other service starts up - they would have to hold up the start of the service they will run in. This can be a bit complex, since we haven’t done these in OpenLMIS yet (spring context access might be limited) and there are cases we want to account for, for example we don’t want them to run when running just the single service or in tests - a solution is either use a flag or base that on consul checks, but obviously this can get increasingly complex. These migrations however would be transparent to the implementer.
  1. The easiest option probably - a standalone script that does the data transfer. Ideally the implementer would just have to populate the config file with something like database credentials for the service and then just boot this script up with one command - it would do the necessary data transfers between service databases. If it helps we can provide this script as a docker container. The downside is obviously that the implementer will have do the migration manually. In this case it will probably only be Malawi, but I suppose cross-service migrations are a topic that can come up in the future again.

I spoke briefly with Josh about this on Slack, and we kinda levitate towards the script solution because of simplicity. What does everyone else think?

Regards,

Paweł