Frequent deploys slowing QA down or blocking it

Yes, this is what I’m trying to raise and I wonder how we can resolve this. Maybe we should stay with many deploy-to-test jobs and change them so only one service is deployed without restarting whole OpenLMIS.


SolDevelo
Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

···

On Mon, Dec 11, 2017 at 3:29 PM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:


SolDevelo
Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Probably yes, but what in situation where I have to modify 10 services? Currently the Jenkins only handle 3 jobs in the same time (also there could be a gap because sometimes a slave instance have to be created) and for some type of jobs only one can be executed (like deploy, performance). In this situation not all docker images will be created before deploy.

Also I don’t think is correct to think like: this pipeline is not completed but the test server will probably have new version of image. In my opinion, If service pipeline is not completed, the service should not be deployed on the test server because what should we do in situation where a new version of service was created, deployed on test server but then related contract-tests fails?

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Mon, Dec 11, 2017 at 3:21 PM, Paweł Albecki palbecki@soldevelo.com wrote:

You said yourself that “referencedata-service pipeline will execute the deploy job before requisition-service pipeline will be completed”. So is that not possible that requisition service is built to image and is deployed with deploy-all-job before requisition contract tests finish?

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

On Mon, Dec 11, 2017 at 3:05 PM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:

I this this is handled by jenkins so if any job in pipeline fails then the deploy job is not executed: Pipeline #1848 <- I hope this pipeline will still be visible.

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Mon, Dec 11, 2017 at 3:02 PM, Paweł Albecki palbecki@soldevelo.com wrote:

This is actually good point, currently our job for deploy all services doesn’t wait for contract test to pass. It’s enough for him that image is built. We will have to modify this somehow, so Jenkins job pull from Docker Hub image only if contract tests pass for given service.

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

On Mon, Dec 11, 2017 at 2:55 PM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:

Yes the will be a single deploy job but before this job we have a lot of other jobs for instance: reference-data-service, reference-data-erd-generation, reference-data-sonar, requisition-service, etc. So it is possible that the referencedata-service pipeline will execute the deploy job before requisition-service pipeline will be completed for example because contract tests for requisition takes longer than the same type of tests for reference-data.

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Mon, Dec 11, 2017 at 2:47 PM, Paweł Albecki palbecki@soldevelo.com wrote:

How so? I thought we are talking about one deploy at hour, so all that developer needs to do is push all changes before the bell tolls.

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

On Mon, Dec 11, 2017 at 2:33 PM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:

Even if a developer push changes from different repositories at the same time, jobs will be executed in different times and basically it is impossible to achieve it.

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Mon, Dec 11, 2017 at 2:07 PM, Paweł Albecki palbecki@soldevelo.com wrote:

If now we change the endpoint in that way the schema will be modified and other services will not be able to handle the new schema - like move endpoint to pageable version. This will probably make the system unavailable, the QA will be blocked and we are not be able to do anything until to the next deploy to the test server which will fix the issue. ’

Is it not possible to push all changes at the same time so they will be deployed once?

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

On Mon, Dec 11, 2017 at 1:26 PM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:

Nick It is easy to imagine a situation where we modify the existing endpoint that is used in many places by different services (e.g. facility endpoints). If now we change the endpoint in that way the schema will be modified and other services will not be able to handle the new schema - like move endpoint to pageable version. This will probably make the system unavailable, the QA will be blocked and we are not be able to do anything until to the next deploy to the test server which will fix the issue. Probably there will be another job to manually do the deploy (without any delay) but I think it would be used to many time because if it does not have any delay so why I should wait.

I think a separate server for QA (handled only by the QA) is a very good idea and I hope it will speed up the QA process. I am only afraid that we could have too many servers to maintain. From what I know now we have four: test server, UAT, perftest server, demo server. Maybe instead of creating a new one we could use the UAT?

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/CAAdp53yahvG1LmcpbyM64PLJxBy2VnP_oJhscEoh5zRwTG3D0g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Mon, Dec 11, 2017 at 1:12 PM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:

From my point of view it would be better to create contract tests in separate tickets to avoid situations where we create tests that only check the happy path.

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Mon, Dec 11, 2017 at 11:46 AM, Paweł Albecki palbecki@soldevelo.com wrote:

I agree that we should make more use of our existing test infrastructure. We definitely need more contract tests which wasn’t added for new features for a while. From my experience I can say lack of them is one of the most frequent reason that test server is “blocked”. The question is when such tests should be added (during work on new feature or in separate ticket created by QA?) and how to enforce that they will be sufficient.
Regarding integration and component tests, do even really have the latter? I feel like in IT Gradle task we only have integration tests and only some of them meet definition of component test (from RTD).

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/CAAJzpfksNCfPR19v8Ck3w5gLjgs9o2V3K%2B0C94zm3CPXFYvquQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Regards,

Paweł

On Thu, Dec 7, 2017 at 12:58 PM, Josh Zamor josh.zamor@villagereach.org wrote:

I agree with many of the thoughts here. test.openlmis.org doesn’t quite fit the QA need, and we should look at using UAT in the meantime for QA purposes. If that’s not possible, we need a separate qa.openlmis.org. This server’s re-deployment would be owned by QA. There is still a need for multiple QA environments, notably when doing team-wide regression testing, however we can still wait on that.

There is also a desire I’m hearing from QA to be able to deploy to this QA server finished tickets. We would need a mechanism such as GitHub Flow or feature flags to be able to do something like that, and indeed this is similar to our need for having a Git flow approach which we think we need for a more stable release process. That said, those are tools we don’t have today. A tool which we do have today is our CI and CD approach, which can very well help with in-process tickets effecting the stability of the SNAPSHOT versions QA is testing. CI and CD however rely on good automated testing - enough coverage and testing the right things. When was the last time we added a new contract test? Why don’t our unit / integration / component tests catch more and provide us early feedback? While we work on the other tools (Git flow, GitHub flow, etc), I’d like to encourage us to remember the importance of our existing test infrastructure. We already have that tool and it can help us in many different ways, QA testing included.

That’s what I’d encourage. As for immediate next step on my end, I’ll be sure to bring this up to Team ILL for the UAT server.

Best,
Josh

On Thursday, December 7, 2017 at 3:53:02 AM UTC-5, jbebak@soldevelo.com wrote:

  I'm not sure if this would be a feasible and a good solution in this project but to me as a tester, it would be best e.g. to use UAT more as a test server (for the changes concerning finished tickets to be deployed), and e.g. to use the test server more as a dev server, i.e. for the deployment of changes that e.g. make some of the tests fail and still need to be worked on. This is only my suggestion, of course.

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/8d984066-8da8-4bcd-b89f-bc8b51a4c5e6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

According to the slack channel the QA will use the UAT server. I hope this will resolve some issues and speed up the testing.


SolDevelo
Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

···

On Mon, Dec 11, 2017 at 4:33 PM, Paweł Albecki palbecki@soldevelo.com wrote:

Yes, this is what I’m trying to raise and I wonder how we can resolve this. Maybe we should stay with many deploy-to-test jobs and change them so only one service is deployed without restarting whole OpenLMIS.

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Mon, Dec 11, 2017 at 3:29 PM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

Probably yes, but what in situation where I have to modify 10 services? Currently the Jenkins only handle 3 jobs in the same time (also there could be a gap because sometimes a slave instance have to be created) and for some type of jobs only one can be executed (like deploy, performance). In this situation not all docker images will be created before deploy.

Also I don’t think is correct to think like: this pipeline is not completed but the test server will probably have new version of image. In my opinion, If service pipeline is not completed, the service should not be deployed on the test server because what should we do in situation where a new version of service was created, deployed on test server but then related contract-tests fails?

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Mon, Dec 11, 2017 at 3:21 PM, Paweł Albecki palbecki@soldevelo.com wrote:

You said yourself that “referencedata-service pipeline will execute the deploy job before requisition-service pipeline will be completed”. So is that not possible that requisition service is built to image and is deployed with deploy-all-job before requisition contract tests finish?

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

On Mon, Dec 11, 2017 at 3:05 PM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:

I this this is handled by jenkins so if any job in pipeline fails then the deploy job is not executed: Pipeline #1848 <- I hope this pipeline will still be visible.

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Mon, Dec 11, 2017 at 3:02 PM, Paweł Albecki palbecki@soldevelo.com wrote:

This is actually good point, currently our job for deploy all services doesn’t wait for contract test to pass. It’s enough for him that image is built. We will have to modify this somehow, so Jenkins job pull from Docker Hub image only if contract tests pass for given service.

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

On Mon, Dec 11, 2017 at 2:55 PM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:

Yes the will be a single deploy job but before this job we have a lot of other jobs for instance: reference-data-service, reference-data-erd-generation, reference-data-sonar, requisition-service, etc. So it is possible that the referencedata-service pipeline will execute the deploy job before requisition-service pipeline will be completed for example because contract tests for requisition takes longer than the same type of tests for reference-data.

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Mon, Dec 11, 2017 at 2:47 PM, Paweł Albecki palbecki@soldevelo.com wrote:

How so? I thought we are talking about one deploy at hour, so all that developer needs to do is push all changes before the bell tolls.

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

On Mon, Dec 11, 2017 at 2:33 PM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:

Even if a developer push changes from different repositories at the same time, jobs will be executed in different times and basically it is impossible to achieve it.

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Mon, Dec 11, 2017 at 2:07 PM, Paweł Albecki palbecki@soldevelo.com wrote:

If now we change the endpoint in that way the schema will be modified and other services will not be able to handle the new schema - like move endpoint to pageable version. This will probably make the system unavailable, the QA will be blocked and we are not be able to do anything until to the next deploy to the test server which will fix the issue. ’

Is it not possible to push all changes at the same time so they will be deployed once?

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

On Mon, Dec 11, 2017 at 1:26 PM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:

Nick It is easy to imagine a situation where we modify the existing endpoint that is used in many places by different services (e.g. facility endpoints). If now we change the endpoint in that way the schema will be modified and other services will not be able to handle the new schema - like move endpoint to pageable version. This will probably make the system unavailable, the QA will be blocked and we are not be able to do anything until to the next deploy to the test server which will fix the issue. Probably there will be another job to manually do the deploy (without any delay) but I think it would be used to many time because if it does not have any delay so why I should wait.

I think a separate server for QA (handled only by the QA) is a very good idea and I hope it will speed up the QA process. I am only afraid that we could have too many servers to maintain. From what I know now we have four: test server, UAT, perftest server, demo server. Maybe instead of creating a new one we could use the UAT?

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/CAAdp53yahvG1LmcpbyM64PLJxBy2VnP_oJhscEoh5zRwTG3D0g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Mon, Dec 11, 2017 at 1:12 PM, Łukasz Lewczyński llewczynski@soldevelo.com wrote:

From my point of view it would be better to create contract tests in separate tickets to avoid situations where we create tests that only check the happy path.

Łukasz Lewczyński
Software Developer
llewczynski@soldevelo.com

On Mon, Dec 11, 2017 at 11:46 AM, Paweł Albecki palbecki@soldevelo.com wrote:

I agree that we should make more use of our existing test infrastructure. We definitely need more contract tests which wasn’t added for new features for a while. From my experience I can say lack of them is one of the most frequent reason that test server is “blocked”. The question is when such tests should be added (during work on new feature or in separate ticket created by QA?) and how to enforce that they will be sufficient.
Regarding integration and component tests, do even really have the latter? I feel like in IT Gradle task we only have integration tests and only some of them meet definition of component test (from RTD).

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/CAAJzpfksNCfPR19v8Ck3w5gLjgs9o2V3K%2B0C94zm3CPXFYvquQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Regards,

Paweł

On Thu, Dec 7, 2017 at 12:58 PM, Josh Zamor josh.zamor@villagereach.org wrote:

I agree with many of the thoughts here. test.openlmis.org doesn’t quite fit the QA need, and we should look at using UAT in the meantime for QA purposes. If that’s not possible, we need a separate qa.openlmis.org. This server’s re-deployment would be owned by QA. There is still a need for multiple QA environments, notably when doing team-wide regression testing, however we can still wait on that.

There is also a desire I’m hearing from QA to be able to deploy to this QA server finished tickets. We would need a mechanism such as GitHub Flow or feature flags to be able to do something like that, and indeed this is similar to our need for having a Git flow approach which we think we need for a more stable release process. That said, those are tools we don’t have today. A tool which we do have today is our CI and CD approach, which can very well help with in-process tickets effecting the stability of the SNAPSHOT versions QA is testing. CI and CD however rely on good automated testing - enough coverage and testing the right things. When was the last time we added a new contract test? Why don’t our unit / integration / component tests catch more and provide us early feedback? While we work on the other tools (Git flow, GitHub flow, etc), I’d like to encourage us to remember the importance of our existing test infrastructure. We already have that tool and it can help us in many different ways, QA testing included.

That’s what I’d encourage. As for immediate next step on my end, I’ll be sure to bring this up to Team ILL for the UAT server.

Best,
Josh

On Thursday, December 7, 2017 at 3:53:02 AM UTC-5, jbebak@soldevelo.com wrote:

  I'm not sure if this would be a feasible and a good solution in this project but to me as a tester, it would be best e.g. to use UAT more as a test server (for the changes concerning finished tickets to be deployed), and e.g. to use the test server more as a dev server, i.e. for the deployment of changes that e.g. make some of the tests fail and still need to be worked on. This is only my suggestion, of course.

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/8d984066-8da8-4bcd-b89f-bc8b51a4c5e6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

On December 12, Sam posted the following message on the #qa Slack channel:

@here Re: Dev forum discussion about testing environment “Frequent deploys slowing QA down or blocking it”
We would like to come to an agreement on which environment to use for testing. Since uat.openlmis.org is not used for demos anymore (we demo in demo-v3.openlmis.org), we would like to have the test team (@sammieim, @jbebak) use uat instead of test.openlmis.org so that we can avoid QA downtime. I would like to hear back from the teams if anyone disagrees. Otherwise we will move forward with uat.

  • If anyone is using uat for any other purposes, please let @joshzamor, @jbebak and myself know
  • Deployment to uat will be scheduled. Should this be hourly or a different schedule? Please provide feedback here.
  • @joshzamor will notify the channel here when the environment is ready (deploy jobs scheduled and running)

Yet nobody responded to this message up until now. Do any of you have any feedback as far as the above ideas are concerned? Has any progress been made with resolving the frequent deploys issue? As far as I can see, UAT still doesn’t have the latest changes, so it can’t be used for testing.

···

On Tuesday, December 12, 2017 at 9:19:25 AM UTC+1, Łukasz Lewczyński wrote:

According to the slack channel the QA will use the UAT server. I hope this will resolve some issues and speed up the testing.

Łukasz Lewczyński
Software Developer
llewc...@soldevelo.com

On Mon, Dec 11, 2017 at 4:33 PM, Paweł Albecki palb...@soldevelo.com wrote:

Yes, this is what I’m trying to raise and I wonder how we can resolve this. Maybe we should stay with many deploy-to-test jobs and change them so only one service is deployed without restarting whole OpenLMIS.

On Mon, Dec 11, 2017 at 3:29 PM, Łukasz Lewczyński llewc...@soldevelo.com wrote:

Probably yes, but what in situation where I have to modify 10 services? Currently the Jenkins only handle 3 jobs in the same time (also there could be a gap because sometimes a slave instance have to be created) and for some type of jobs only one can be executed (like deploy, performance). In this situation not all docker images will be created before deploy.

Also I don’t think is correct to think like: this pipeline is not completed but the test server will probably have new version of image. In my opinion, If service pipeline is not completed, the service should not be deployed on the test server because what should we do in situation where a new version of service was created, deployed on test server but then related contract-tests fails?

Łukasz Lewczyński
Software Developer
llewc...@soldevelo.com

On Mon, Dec 11, 2017 at 3:21 PM, Paweł Albecki palb...@soldevelo.com wrote:

You said yourself that “referencedata-service pipeline will execute the deploy job before requisition-service pipeline will be completed”. So is that not possible that requisition service is built to image and is deployed with deploy-all-job before requisition contract tests finish?

On Mon, Dec 11, 2017 at 3:05 PM, Łukasz Lewczyński llewc...@soldevelo.com wrote:

I this this is handled by jenkins so if any job in pipeline fails then the deploy job is not executed: Pipeline #1848 <- I hope this pipeline will still be visible.

Łukasz Lewczyński
Software Developer
llewc...@soldevelo.com

On Mon, Dec 11, 2017 at 3:02 PM, Paweł Albecki palb...@soldevelo.com wrote:

This is actually good point, currently our job for deploy all services doesn’t wait for contract test to pass. It’s enough for him that image is built. We will have to modify this somehow, so Jenkins job pull from Docker Hub image only if contract tests pass for given service.

On Mon, Dec 11, 2017 at 2:55 PM, Łukasz Lewczyński llewc...@soldevelo.com wrote:

Yes the will be a single deploy job but before this job we have a lot of other jobs for instance: reference-data-service, reference-data-erd-generation, reference-data-sonar, requisition-service, etc. So it is possible that the referencedata-service pipeline will execute the deploy job before requisition-service pipeline will be completed for example because contract tests for requisition takes longer than the same type of tests for reference-data.

Łukasz Lewczyński
Software Developer
llewc...@soldevelo.com

On Mon, Dec 11, 2017 at 2:47 PM, Paweł Albecki palb...@soldevelo.com wrote:

How so? I thought we are talking about one deploy at hour, so all that developer needs to do is push all changes before the bell tolls.

On Mon, Dec 11, 2017 at 2:33 PM, Łukasz Lewczyński llewc...@soldevelo.com wrote:

Even if a developer push changes from different repositories at the same time, jobs will be executed in different times and basically it is impossible to achieve it.

Łukasz Lewczyński
Software Developer
llewc...@soldevelo.com

On Mon, Dec 11, 2017 at 2:07 PM, Paweł Albecki palb...@soldevelo.com wrote:

If now we change the endpoint in that way the schema will be modified and other services will not be able to handle the new schema - like move endpoint to pageable version. This will probably make the system unavailable, the QA will be blocked and we are not be able to do anything until to the next deploy to the test server which will fix the issue. ’

Is it not possible to push all changes at the same time so they will be deployed once?

On Mon, Dec 11, 2017 at 1:26 PM, Łukasz Lewczyński llewc...@soldevelo.com wrote:

Nick It is easy to imagine a situation where we modify the existing endpoint that is used in many places by different services (e.g. facility endpoints). If now we change the endpoint in that way the schema will be modified and other services will not be able to handle the new schema - like move endpoint to pageable version. This will probably make the system unavailable, the QA will be blocked and we are not be able to do anything until to the next deploy to the test server which will fix the issue. Probably there will be another job to manually do the deploy (without any delay) but I think it would be used to many time because if it does not have any delay so why I should wait.

I think a separate server for QA (handled only by the QA) is a very good idea and I hope it will speed up the QA process. I am only afraid that we could have too many servers to maintain. From what I know now we have four: test server, UAT, perftest server, demo server. Maybe instead of creating a new one we could use the UAT?

Łukasz Lewczyński
Software Developer
llewc...@soldevelo.com

On Mon, Dec 11, 2017 at 1:12 PM, Łukasz Lewczyński llewc...@soldevelo.com wrote:

From my point of view it would be better to create contract tests in separate tickets to avoid situations where we create tests that only check the happy path.

Łukasz Lewczyński
Software Developer
llewc...@soldevelo.com

On Mon, Dec 11, 2017 at 11:46 AM, Paweł Albecki palb...@soldevelo.com wrote:

I agree that we should make more use of our existing test infrastructure. We definitely need more contract tests which wasn’t added for new features for a while. From my experience I can say lack of them is one of the most frequent reason that test server is “blocked”. The question is when such tests should be added (during work on new feature or in separate ticket created by QA?) and how to enforce that they will be sufficient.
Regarding integration and component tests, do even really have the latter? I feel like in IT Gradle task we only have integration tests and only some of them meet definition of component test (from RTD).

Regards,

Paweł

On Thu, Dec 7, 2017 at 12:58 PM, Josh Zamor josh....@villagereach.org wrote:

I agree with many of the thoughts here. test.openlmis.org doesn’t quite fit the QA need, and we should look at using UAT in the meantime for QA purposes. If that’s not possible, we need a separate qa.openlmis.org. This server’s re-deployment would be owned by QA. There is still a need for multiple QA environments, notably when doing team-wide regression testing, however we can still wait on that.

There is also a desire I’m hearing from QA to be able to deploy to this QA server finished tickets. We would need a mechanism such as GitHub Flow or feature flags to be able to do something like that, and indeed this is similar to our need for having a Git flow approach which we think we need for a more stable release process. That said, those are tools we don’t have today. A tool which we do have today is our CI and CD approach, which can very well help with in-process tickets effecting the stability of the SNAPSHOT versions QA is testing. CI and CD however rely on good automated testing - enough coverage and testing the right things. When was the last time we added a new contract test? Why don’t our unit / integration / component tests catch more and provide us early feedback? While we work on the other tools (Git flow, GitHub flow, etc), I’d like to encourage us to remember the importance of our existing test infrastructure. We already have that tool and it can help us in many different ways, QA testing included.

That’s what I’d encourage. As for immediate next step on my end, I’ll be sure to bring this up to Team ILL for the UAT server.

Best,
Josh

On Thursday, December 7, 2017 at 3:53:02 AM UTC-5, jbe...@soldevelo.com wrote:

  I'm not sure if this would be a feasible and a good solution in this project but to me as a tester, it would be best e.g. to use UAT more as a test server (for the changes concerning finished tickets to be deployed), and e.g. to use the test server more as a dev server, i.e. for the deployment of changes that e.g. make some of the tests fail and still need to be worked on. This is only my suggestion, of course.

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.

To post to this group, send email to openlm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/8d984066-8da8-4bcd-b89f-bc8b51a4c5e6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Albecki

    Software Developer

     palb...@soldevelo.com

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.

To post to this group, send email to openlm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/CAAJzpfksNCfPR19v8Ck3w5gLjgs9o2V3K%2B0C94zm3CPXFYvquQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.

To post to this group, send email to openlm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/CAAdp53yahvG1LmcpbyM64PLJxBy2VnP_oJhscEoh5zRwTG3D0g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Albecki

    Software Developer

     palb...@soldevelo.com

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Paweł Albecki

    Software Developer

     palb...@soldevelo.com

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Paweł Albecki

    Software Developer

     palb...@soldevelo.com

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Paweł Albecki

    Software Developer

     palb...@soldevelo.com

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Paweł Albecki

    Software Developer

     palb...@soldevelo.com

**
SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41


SolDevelo
Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

It took longer than we thought as there was a blocking issue that needed to be resolved first, however the UAT instance is now ready for QA to use it:

Thanks for bringing this up Joanna, please take a look at http://uat.openlmis.org and let us know if this new approach helps your process.

Best,

Josh

···

On Wednesday, December 6, 2017 at 8:27:00 AM UTC-6, jbebak@soldevelo.com wrote:

I would like to mention again the issue that had, to my knowledge, been already discussed several times but still, no solution to it was found. One has to perform tests with the use of the most recent changes. It takes some time for the changes to appear on the test and performance servers, it is also impossible to test for some time after the server restarts because of the Bad Gateway error. When changes are made frequently, it can block testing for a really long time. There are situations in which for this reason the testing of a ticket which normally should take up to 20 minutes can take even two hours (e.g. this is what happened today). I think something needs to be done about it, as it can considerably slow the work down.

After a whole sprint with the new solution, I can say that moving testing to UAT and scheduling once-an-hour deploys has helped me greatly. I am not slowed down or blocked by deploys anymore, so thank you very much for implementing this solution.

···

On Saturday, December 30, 2017 at 12:11:28 AM UTC+1, Josh Zamor wrote:

It took longer than we thought as there was a blocking issue that needed to be resolved first, however the UAT instance is now ready for QA to use it:

Thanks for bringing this up Joanna, please take a look at http://uat.openlmis.org and let us know if this new approach helps your process.

Best,

Josh

On Wednesday, December 6, 2017 at 8:27:00 AM UTC-6, jbe...@soldevelo.com wrote:

I would like to mention again the issue that had, to my knowledge, been already discussed several times but still, no solution to it was found. One has to perform tests with the use of the most recent changes. It takes some time for the changes to appear on the test and performance servers, it is also impossible to test for some time after the server restarts because of the Bad Gateway error. When changes are made frequently, it can block testing for a really long time. There are situations in which for this reason the testing of a ticket which normally should take up to 20 minutes can take even two hours (e.g. this is what happened today). I think something needs to be done about it, as it can considerably slow the work down.