Faulty migrations?

Technical Committee
#1

In the past couple weeks an issue surfaced that our automated database migration test job wasn’t fully capturing migrations that weren’t production ready. Could someone(s) summarize what the overall issue and steps taken to fix have been? I’ve seen some of this in Slack, however this is critical for our release so I’d prefer if we could get it written down here on the technical forum.

One of the minor things I’ve noticed is that we’ve opened up port 8500 (Consul’s management port) to the Docker host in order for the migration tests to run - this is enough of a security concern that if we shipped this, we’d want to ensure we had documentation for how to not expose this port beyond the docker host. I don’t want this issue to dominate this discussion, it’s minor, however lets include here changes like this to make the tests run.

Finally it’d help if people expressed their confidence in the migrations we’ve written since Ref Distro v3.2.1. The automated tests, if working, would still be dependent on demo data, which in the last showcase it appeared we might need more. Are we in better shape? Similar shape?

Best,
Josh

#2

As you said I opened up port 8500 and add check that service list in the migration test script is not empty. After those changes migration test failed so I thought that everything is okay but Pawel Albecki found out that this error was connected with javers rather than migrations.

I think we need more demo data - each table should contain some data with all variations of some fields (if SQL field is related with java enum field, the SQL field should contain all possibilities of enum). Also I think we could create some integration test that verify our migrations. A good example is in the reference data service: migration test.

Regards,
Lukasz


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

···

On Wed, Mar 7, 2018 at 8:25 PM, josh.zamor@openlmis.org wrote:

In the past couple weeks an issue surfaced that our automated database migration test job wasn’t fully capturing migrations that weren’t production ready. Could someone(s) summarize what the overall issue and steps taken to fix have been? I’ve seen some of this in Slack, however this is critical for our release so I’d prefer if we could get it written down here on the technical forum.

One of the minor things I’ve noticed is that we’ve opened up port 8500 (Consul’s management port) to the Docker host in order for the migration tests to run - this is enough of a security concern that if we shipped this, we’d want to ensure we had documentation for how to not expose this port beyond the docker host. I don’t want this issue to dominate this discussion, it’s minor, however lets include here changes like this to make the tests run.

Finally it’d help if people expressed their confidence in the migrations we’ve written since Ref Distro v3.2.1. The automated tests, if working, would still be dependent on demo data, which in the last showcase it appeared we might need more. Are we in better shape? Similar shape?

Best,
Josh

–

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/37e6e39d-bb17-45c6-8b04-d136de53dcc1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

#3

Thanks Lukasz,

So on the port topic I take it that we could take that off the host mount? This line: https://github.com/OpenLMIS/openlmis-ref-distro/blob/master/docker-compose.yml#L10

Would you be able to do that if that’s so Lukasz?

On the broader topic of our migration tests, are they working now?

Best,
Josh

···

On Thursday, March 8, 2018 at 12:24:10 AM UTC-8, Łukasz Lewczyński wrote:

As you said I opened up port 8500 and add check that service list in the migration test script is not empty. After those changes migration test failed so I thought that everything is okay but Pawel Albecki found out that this error was connected with javers rather than migrations.

I think we need more demo data - each table should contain some data with all variations of some fields (if SQL field is related with java enum field, the SQL field should contain all possibilities of enum). Also I think we could create some integration test that verify our migrations. A good example is in the reference data service: migration test.

Regards,
Lukasz

Łukasz Lewczyński
Software Developer
llewc...@soldevelo.com

On Wed, Mar 7, 2018 at 8:25 PM, josh....@openlmis.org wrote:

In the past couple weeks an issue surfaced that our automated database migration test job wasn’t fully capturing migrations that weren’t production ready. Could someone(s) summarize what the overall issue and steps taken to fix have been? I’ve seen some of this in Slack, however this is critical for our release so I’d prefer if we could get it written down here on the technical forum.

One of the minor things I’ve noticed is that we’ve opened up port 8500 (Consul’s management port) to the Docker host in order for the migration tests to run - this is enough of a security concern that if we shipped this, we’d want to ensure we had documentation for how to not expose this port beyond the docker host. I don’t want this issue to dominate this discussion, it’s minor, however lets include here changes like this to make the tests run.

Finally it’d help if people expressed their confidence in the migrations we’ve written since Ref Distro v3.2.1. The automated tests, if working, would still be dependent on demo data, which in the last showcase it appeared we might need more. Are we in better shape? Similar shape?

Best,
Josh

–

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.

To post to this group, send email to openlm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/37e6e39d-bb17-45c6-8b04-d136de53dcc1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

#4

Oh and not to lose the point about those Flyway tests: I like that approach looking at the basics. Would someone be willing to present on how it works, if it should be expanded to other services, and possibly even replace the CI Jenkins job on the next technical committee call?

Josh

···

On Wednesday, March 7, 2018 at 11:25:59 AM UTC-8, josh....@openlmis.org wrote:

In the past couple weeks an issue surfaced that our automated database migration test job wasn’t fully capturing migrations that weren’t production ready. Could someone(s) summarize what the overall issue and steps taken to fix have been? I’ve seen some of this in Slack, however this is critical for our release so I’d prefer if we could get it written down here on the technical forum.

One of the minor things I’ve noticed is that we’ve opened up port 8500 (Consul’s management port) to the Docker host in order for the migration tests to run - this is enough of a security concern that if we shipped this, we’d want to ensure we had documentation for how to not expose this port beyond the docker host. I don’t want this issue to dominate this discussion, it’s minor, however lets include here changes like this to make the tests run.

Finally it’d help if people expressed their confidence in the migrations we’ve written since Ref Distro v3.2.1. The automated tests, if working, would still be dependent on demo data, which in the last showcase it appeared we might need more. Are we in better shape? Similar shape?

Best,
Josh

#5

I am not sure if I follow. I should restore previous version of the line https://github.com/OpenLMIS/openlmis-ref-distro/blob/master/docker-compose.yml#L10 ? If I do this, the migration test will not be able to retrieve a list of services that are in the newest version of OpenLMIS. I could take a look on those Flyway tests.

Pawel: If you have any useful resources about those tests (from what I see you have created them) could you bring them to this topic?

Regards,

Lukasz


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

···

On Tue, Mar 13, 2018 at 6:45 AM, josh.zamor@openlmis.org wrote:

Oh and not to lose the point about those Flyway tests: I like that approach looking at the basics. Would someone be willing to present on how it works, if it should be expanded to other services, and possibly even replace the CI Jenkins job on the next technical committee call?

Josh

On Wednesday, March 7, 2018 at 11:25:59 AM UTC-8, josh....@openlmis.org wrote:

In the past couple weeks an issue surfaced that our automated database migration test job wasn’t fully capturing migrations that weren’t production ready. Could someone(s) summarize what the overall issue and steps taken to fix have been? I’ve seen some of this in Slack, however this is critical for our release so I’d prefer if we could get it written down here on the technical forum.

One of the minor things I’ve noticed is that we’ve opened up port 8500 (Consul’s management port) to the Docker host in order for the migration tests to run - this is enough of a security concern that if we shipped this, we’d want to ensure we had documentation for how to not expose this port beyond the docker host. I don’t want this issue to dominate this discussion, it’s minor, however lets include here changes like this to make the tests run.

Finally it’d help if people expressed their confidence in the migrations we’ve written since Ref Distro v3.2.1. The automated tests, if working, would still be dependent on demo data, which in the last showcase it appeared we might need more. Are we in better shape? Similar shape?

Best,
Josh

–

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/128cbbfb-e921-4dd2-a82e-f1e1e15c5dc8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

#6

These tests were not positively reviewed when I created them. The general migrations from migration-test were favored. I can briefly present how it works on next technical committee call but I doubt it could replace the CI Jenkins job. It can be used to supplement migration-test though.

Regards,

Paweł


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

···

On Tue, Mar 13, 2018 at 9:27 AM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:

I am not sure if I follow. I should restore previous version of the line https://github.com/OpenLMIS/openlmis-ref-distro/blob/master/docker-compose.yml#L10 ? If I do this, the migration test will not be able to retrieve a list of services that are in the newest version of OpenLMIS. I could take a look on those Flyway tests.

Pawel: If you have any useful resources about those tests (from what I see you have created them) could you bring them to this topic?

Regards,

Lukasz


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

–

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Tue, Mar 13, 2018 at 6:45 AM, josh.zamor@openlmis.org wrote:

Oh and not to lose the point about those Flyway tests: I like that approach looking at the basics. Would someone be willing to present on how it works, if it should be expanded to other services, and possibly even replace the CI Jenkins job on the next technical committee call?

Josh

On Wednesday, March 7, 2018 at 11:25:59 AM UTC-8, josh....@openlmis.org wrote:

In the past couple weeks an issue surfaced that our automated database migration test job wasn’t fully capturing migrations that weren’t production ready. Could someone(s) summarize what the overall issue and steps taken to fix have been? I’ve seen some of this in Slack, however this is critical for our release so I’d prefer if we could get it written down here on the technical forum.

One of the minor things I’ve noticed is that we’ve opened up port 8500 (Consul’s management port) to the Docker host in order for the migration tests to run - this is enough of a security concern that if we shipped this, we’d want to ensure we had documentation for how to not expose this port beyond the docker host. I don’t want this issue to dominate this discussion, it’s minor, however lets include here changes like this to make the tests run.

Finally it’d help if people expressed their confidence in the migrations we’ve written since Ref Distro v3.2.1. The automated tests, if working, would still be dependent on demo data, which in the last showcase it appeared we might need more. Are we in better shape? Similar shape?

Best,
Josh

–

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/128cbbfb-e921-4dd2-a82e-f1e1e15c5dc8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com
#7

Regarding the port, can’t we just remove it from the ref distro, and make migration tests use a docker-compose.override.yml? The override in migration tests can open the 8500 port and we won’t have that line in the ref distro.

I’m actually not sure what the issue was, but the idea of migration tests was to just look at the logs for any failures? Was that approach not sufficient and we had to resort to connecting to Consul?

Regards,

Paweł


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

···

On Tue, Mar 13, 2018 at 11:15 AM, Paweł Albecki palbecki@soldevelo.com wrote:

Regards,

Paweł

These tests were not positively reviewed when I created them. The general migrations from migration-test were favored. I can briefly present how it works on next technical committee call but I doubt it could replace the CI Jenkins job. It can be used to supplement migration-test though.


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/CAAJzpfkwmqgbBx44QNkPqw9mtRwUBkTAaOuNXNk8%3DGvO6dz7DA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

–

On Tue, Mar 13, 2018 at 9:27 AM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:

I am not sure if I follow. I should restore previous version of the line https://github.com/OpenLMIS/openlmis-ref-distro/blob/master/docker-compose.yml#L10 ? If I do this, the migration test will not be able to retrieve a list of services that are in the newest version of OpenLMIS. I could take a look on those Flyway tests.

Pawel: If you have any useful resources about those tests (from what I see you have created them) could you bring them to this topic?

Regards,

Lukasz


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

–

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Tue, Mar 13, 2018 at 6:45 AM, josh.zamor@openlmis.org wrote:

Oh and not to lose the point about those Flyway tests: I like that approach looking at the basics. Would someone be willing to present on how it works, if it should be expanded to other services, and possibly even replace the CI Jenkins job on the next technical committee call?

Josh

On Wednesday, March 7, 2018 at 11:25:59 AM UTC-8, josh....@openlmis.org wrote:

In the past couple weeks an issue surfaced that our automated database migration test job wasn’t fully capturing migrations that weren’t production ready. Could someone(s) summarize what the overall issue and steps taken to fix have been? I’ve seen some of this in Slack, however this is critical for our release so I’d prefer if we could get it written down here on the technical forum.

One of the minor things I’ve noticed is that we’ve opened up port 8500 (Consul’s management port) to the Docker host in order for the migration tests to run - this is enough of a security concern that if we shipped this, we’d want to ensure we had documentation for how to not expose this port beyond the docker host. I don’t want this issue to dominate this discussion, it’s minor, however lets include here changes like this to make the tests run.

Finally it’d help if people expressed their confidence in the migrations we’ve written since Ref Distro v3.2.1. The automated tests, if working, would still be dependent on demo data, which in the last showcase it appeared we might need more. Are we in better shape? Similar shape?

Best,
Josh

–

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/128cbbfb-e921-4dd2-a82e-f1e1e15c5dc8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Gesek

    Technical Project Manager

     pgesek@soldevelo.com / +48 690 020 875
#8

I found one issue with migrations in the reference data. We added constraint on the supply_lines table that there could be only one pair of supervisoryNodeId and programId. I use 9.0.1 demo data for migration tests and some of them does not match the new requirement. Should we handle this situation somehow? Do implementators are responsible to ensure that data in this table are correct before they migrate to OpenLMIS 3.3? If yes, we probably need to add this information to release note.


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

···

On Tue, Mar 13, 2018 at 4:53 PM, Paweł Gesek pgesek@soldevelo.com wrote:

Regarding the port, can’t we just remove it from the ref distro, and make migration tests use a docker-compose.override.yml? The override in migration tests can open the 8500 port and we won’t have that line in the ref distro.

I’m actually not sure what the issue was, but the idea of migration tests was to just look at the logs for any failures? Was that approach not sufficient and we had to resort to connecting to Consul?

Regards,

Paweł

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Tue, Mar 13, 2018 at 11:15 AM, Paweł Albecki palbecki@soldevelo.com wrote:

Regards,

Paweł

These tests were not positively reviewed when I created them. The general migrations from migration-test were favored. I can briefly present how it works on next technical committee call but I doubt it could replace the CI Jenkins job. It can be used to supplement migration-test though.

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/CAAJzpfkwmqgbBx44QNkPqw9mtRwUBkTAaOuNXNk8%3DGvO6dz7DA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Gesek

    Technical Project Manager

     pgesek@soldevelo.com / +48 690 020 875

–

On Tue, Mar 13, 2018 at 9:27 AM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:

I am not sure if I follow. I should restore previous version of the line https://github.com/OpenLMIS/openlmis-ref-distro/blob/master/docker-compose.yml#L10 ? If I do this, the migration test will not be able to retrieve a list of services that are in the newest version of OpenLMIS. I could take a look on those Flyway tests.

Pawel: If you have any useful resources about those tests (from what I see you have created them) could you bring them to this topic?

Regards,

Lukasz

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

–

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Tue, Mar 13, 2018 at 6:45 AM, josh.zamor@openlmis.org wrote:

Oh and not to lose the point about those Flyway tests: I like that approach looking at the basics. Would someone be willing to present on how it works, if it should be expanded to other services, and possibly even replace the CI Jenkins job on the next technical committee call?

Josh

On Wednesday, March 7, 2018 at 11:25:59 AM UTC-8, josh....@openlmis.org wrote:

In the past couple weeks an issue surfaced that our automated database migration test job wasn’t fully capturing migrations that weren’t production ready. Could someone(s) summarize what the overall issue and steps taken to fix have been? I’ve seen some of this in Slack, however this is critical for our release so I’d prefer if we could get it written down here on the technical forum.

One of the minor things I’ve noticed is that we’ve opened up port 8500 (Consul’s management port) to the Docker host in order for the migration tests to run - this is enough of a security concern that if we shipped this, we’d want to ensure we had documentation for how to not expose this port beyond the docker host. I don’t want this issue to dominate this discussion, it’s minor, however lets include here changes like this to make the tests run.

Finally it’d help if people expressed their confidence in the migrations we’ve written since Ref Distro v3.2.1. The automated tests, if working, would still be dependent on demo data, which in the last showcase it appeared we might need more. Are we in better shape? Similar shape?

Best,
Josh

–

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/128cbbfb-e921-4dd2-a82e-f1e1e15c5dc8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

#9

Paweł,

It’s a good idea to just override docker compose file and open the 8500 port using bash script. I can do it, after we resolve more important issues with migration (like Schema-validation: missing column [dismissTimestamp] in table [cce_alerts]).

Regarding the consul usage. We use it to get a list of services so we can wait unit they are started up and serving (if they all do, migration test is successfully exited).


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

···

On Tue, Mar 13, 2018 at 4:53 PM, Paweł Gesek pgesek@soldevelo.com wrote:

Regarding the port, can’t we just remove it from the ref distro, and make migration tests use a docker-compose.override.yml? The override in migration tests can open the 8500 port and we won’t have that line in the ref distro.

I’m actually not sure what the issue was, but the idea of migration tests was to just look at the logs for any failures? Was that approach not sufficient and we had to resort to connecting to Consul?

Regards,

Paweł


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

–

On Tue, Mar 13, 2018 at 11:15 AM, Paweł Albecki palbecki@soldevelo.com wrote:

Regards,

Paweł

These tests were not positively reviewed when I created them. The general migrations from migration-test were favored. I can briefly present how it works on next technical committee call but I doubt it could replace the CI Jenkins job. It can be used to supplement migration-test though.


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/CAAJzpfkwmqgbBx44QNkPqw9mtRwUBkTAaOuNXNk8%3DGvO6dz7DA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Gesek

    Technical Project Manager

     pgesek@soldevelo.com / +48 690 020 875

–

On Tue, Mar 13, 2018 at 9:27 AM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:

I am not sure if I follow. I should restore previous version of the line https://github.com/OpenLMIS/openlmis-ref-distro/blob/master/docker-compose.yml#L10 ? If I do this, the migration test will not be able to retrieve a list of services that are in the newest version of OpenLMIS. I could take a look on those Flyway tests.

Pawel: If you have any useful resources about those tests (from what I see you have created them) could you bring them to this topic?

Regards,

Lukasz


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

–

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Tue, Mar 13, 2018 at 6:45 AM, josh.zamor@openlmis.org wrote:

Oh and not to lose the point about those Flyway tests: I like that approach looking at the basics. Would someone be willing to present on how it works, if it should be expanded to other services, and possibly even replace the CI Jenkins job on the next technical committee call?

Josh

On Wednesday, March 7, 2018 at 11:25:59 AM UTC-8, josh....@openlmis.org wrote:

In the past couple weeks an issue surfaced that our automated database migration test job wasn’t fully capturing migrations that weren’t production ready. Could someone(s) summarize what the overall issue and steps taken to fix have been? I’ve seen some of this in Slack, however this is critical for our release so I’d prefer if we could get it written down here on the technical forum.

One of the minor things I’ve noticed is that we’ve opened up port 8500 (Consul’s management port) to the Docker host in order for the migration tests to run - this is enough of a security concern that if we shipped this, we’d want to ensure we had documentation for how to not expose this port beyond the docker host. I don’t want this issue to dominate this discussion, it’s minor, however lets include here changes like this to make the tests run.

Finally it’d help if people expressed their confidence in the migrations we’ve written since Ref Distro v3.2.1. The automated tests, if working, would still be dependent on demo data, which in the last showcase it appeared we might need more. Are we in better shape? Similar shape?

Best,
Josh

–

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/128cbbfb-e921-4dd2-a82e-f1e1e15c5dc8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com
#10

Just to clarify, I didn’t mean replacing anything with a bash script, I meant the override feature of docker compose:

https://docs.docker.com/compose/extends/

Not sure if that’s what you were referring to.

Regards,

Paweł


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

···

On Wed, Mar 14, 2018 at 11:09 AM, Paweł Albecki palbecki@soldevelo.com wrote:

Paweł,

It’s a good idea to just override docker compose file and open the 8500 port using bash script. I can do it, after we resolve more important issues with migration (like Schema-validation: missing column [dismissTimestamp] in table [cce_alerts]).

Regarding the consul usage. We use it to get a list of services so we can wait unit they are started up and serving (if they all do, migration test is successfully exited).


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

–

On Tue, Mar 13, 2018 at 4:53 PM, Paweł Gesek pgesek@soldevelo.com wrote:

Regarding the port, can’t we just remove it from the ref distro, and make migration tests use a docker-compose.override.yml? The override in migration tests can open the 8500 port and we won’t have that line in the ref distro.

I’m actually not sure what the issue was, but the idea of migration tests was to just look at the logs for any failures? Was that approach not sufficient and we had to resort to connecting to Consul?

Regards,

Paweł


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

–

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

On Tue, Mar 13, 2018 at 11:15 AM, Paweł Albecki palbecki@soldevelo.com wrote:

Regards,

Paweł

These tests were not positively reviewed when I created them. The general migrations from migration-test were favored. I can briefly present how it works on next technical committee call but I doubt it could replace the CI Jenkins job. It can be used to supplement migration-test though.


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/CAAJzpfkwmqgbBx44QNkPqw9mtRwUBkTAaOuNXNk8%3DGvO6dz7DA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Gesek

    Technical Project Manager

     pgesek@soldevelo.com / +48 690 020 875

–

On Tue, Mar 13, 2018 at 9:27 AM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:

I am not sure if I follow. I should restore previous version of the line https://github.com/OpenLMIS/openlmis-ref-distro/blob/master/docker-compose.yml#L10 ? If I do this, the migration test will not be able to retrieve a list of services that are in the newest version of OpenLMIS. I could take a look on those Flyway tests.

Pawel: If you have any useful resources about those tests (from what I see you have created them) could you bring them to this topic?

Regards,

Lukasz


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

–

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Tue, Mar 13, 2018 at 6:45 AM, josh.zamor@openlmis.org wrote:

Oh and not to lose the point about those Flyway tests: I like that approach looking at the basics. Would someone be willing to present on how it works, if it should be expanded to other services, and possibly even replace the CI Jenkins job on the next technical committee call?

Josh

On Wednesday, March 7, 2018 at 11:25:59 AM UTC-8, josh....@openlmis.org wrote:

In the past couple weeks an issue surfaced that our automated database migration test job wasn’t fully capturing migrations that weren’t production ready. Could someone(s) summarize what the overall issue and steps taken to fix have been? I’ve seen some of this in Slack, however this is critical for our release so I’d prefer if we could get it written down here on the technical forum.

One of the minor things I’ve noticed is that we’ve opened up port 8500 (Consul’s management port) to the Docker host in order for the migration tests to run - this is enough of a security concern that if we shipped this, we’d want to ensure we had documentation for how to not expose this port beyond the docker host. I don’t want this issue to dominate this discussion, it’s minor, however lets include here changes like this to make the tests run.

Finally it’d help if people expressed their confidence in the migrations we’ve written since Ref Distro v3.2.1. The automated tests, if working, would still be dependent on demo data, which in the last showcase it appeared we might need more. Are we in better shape? Similar shape?

Best,
Josh

–

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/128cbbfb-e921-4dd2-a82e-f1e1e15c5dc8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Gesek

    Technical Project Manager

     pgesek@soldevelo.com / +48 690 020 875
#11

Hi all again,

I wrote a wiki page about service migration tests. I am looking forward for any comments about it.

https://openlmis.atlassian.net/wiki/spaces/OP/pages/304906282/Service+Migration+Tests

Regards,

Lukasz


SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

···

On Wed, Mar 14, 2018 at 3:21 PM, Paweł Gesek pgesek@soldevelo.com wrote:

Just to clarify, I didn’t mean replacing anything with a bash script, I meant the override feature of docker compose:

https://docs.docker.com/compose/extends/

Not sure if that’s what you were referring to.

Regards,

Paweł

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Wed, Mar 14, 2018 at 11:09 AM, Paweł Albecki palbecki@soldevelo.com wrote:

Paweł,

It’s a good idea to just override docker compose file and open the 8500 port using bash script. I can do it, after we resolve more important issues with migration (like Schema-validation: missing column [dismissTimestamp] in table [cce_alerts]).

Regarding the consul usage. We use it to get a list of services so we can wait unit they are started up and serving (if they all do, migration test is successfully exited).

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

–
Paweł Gesek

    Technical Project Manager

     pgesek@soldevelo.com / +48 690 020 875

On Tue, Mar 13, 2018 at 4:53 PM, Paweł Gesek pgesek@soldevelo.com wrote:

Regarding the port, can’t we just remove it from the ref distro, and make migration tests use a docker-compose.override.yml? The override in migration tests can open the 8500 port and we won’t have that line in the ref distro.

I’m actually not sure what the issue was, but the idea of migration tests was to just look at the logs for any failures? Was that approach not sufficient and we had to resort to connecting to Consul?

Regards,

Paweł

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

–

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

On Tue, Mar 13, 2018 at 11:15 AM, Paweł Albecki palbecki@soldevelo.com wrote:

Regards,

Paweł

These tests were not positively reviewed when I created them. The general migrations from migration-test were favored. I can briefly present how it works on next technical committee call but I doubt it could replace the CI Jenkins job. It can be used to supplement migration-test though.

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/CAAJzpfkwmqgbBx44QNkPqw9mtRwUBkTAaOuNXNk8%3DGvO6dz7DA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Gesek

    Technical Project Manager

     pgesek@soldevelo.com / +48 690 020 875

–

On Tue, Mar 13, 2018 at 9:27 AM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:

I am not sure if I follow. I should restore previous version of the line https://github.com/OpenLMIS/openlmis-ref-distro/blob/master/docker-compose.yml#L10 ? If I do this, the migration test will not be able to retrieve a list of services that are in the newest version of OpenLMIS. I could take a look on those Flyway tests.

Pawel: If you have any useful resources about those tests (from what I see you have created them) could you bring them to this topic?

Regards,

Lukasz

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

–

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Tue, Mar 13, 2018 at 6:45 AM, josh.zamor@openlmis.org wrote:

Oh and not to lose the point about those Flyway tests: I like that approach looking at the basics. Would someone be willing to present on how it works, if it should be expanded to other services, and possibly even replace the CI Jenkins job on the next technical committee call?

Josh

On Wednesday, March 7, 2018 at 11:25:59 AM UTC-8, josh....@openlmis.org wrote:

In the past couple weeks an issue surfaced that our automated database migration test job wasn’t fully capturing migrations that weren’t production ready. Could someone(s) summarize what the overall issue and steps taken to fix have been? I’ve seen some of this in Slack, however this is critical for our release so I’d prefer if we could get it written down here on the technical forum.

One of the minor things I’ve noticed is that we’ve opened up port 8500 (Consul’s management port) to the Docker host in order for the migration tests to run - this is enough of a security concern that if we shipped this, we’d want to ensure we had documentation for how to not expose this port beyond the docker host. I don’t want this issue to dominate this discussion, it’s minor, however lets include here changes like this to make the tests run.

Finally it’d help if people expressed their confidence in the migrations we’ve written since Ref Distro v3.2.1. The automated tests, if working, would still be dependent on demo data, which in the last showcase it appeared we might need more. Are we in better shape? Similar shape?

Best,
Josh

–

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/128cbbfb-e921-4dd2-a82e-f1e1e15c5dc8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.