End to end notification testing?

Hi all,

On one of our Bug triage meetings, Mary Jo asked about automated end-to-end testing around notifications as we encountered few bugs related to notifications during Regression testing. I promised to raise this topic on Dev Forum.

I’ve also done some research. The simplest way for e2e notification testing would be use our contract tests for this purpose. We could make use of some fake SMPT server (e.g. https://github.com/Nilhcem/FakeSMTP and its docker wrapper https://github.com/digiPlant/docker-fake-smtp) that will intercept sent emails and save them to workspace, then test suite would read and check the email receiver, title, content.

I wonder if it is worth an effort? Do you have any experience with such tests? Please share your opinion.

Best regards,

Paweł.

Hi Pawel,

Sounds interesting. It’d help to know caused the bugs. As in do we really need to test that we’re sending appropriately to an SMTP server, or do we just need to test the contract of the Notification service?

Best,
Josh

···

On Tuesday, April 24, 2018 at 8:26:37 AM UTC-7, Paweł Albecki wrote:

Hi all,

On one of our Bug triage meetings, Mary Jo asked about automated end-to-end testing around notifications as we encountered few bugs related to notifications during Regression testing. I promised to raise this topic on Dev Forum.

I’ve also done some research. The simplest way for e2e notification testing would be use our contract tests for this purpose. We could make use of some fake SMPT server (e.g. https://github.com/Nilhcem/FakeSMTP and its docker wrapper https://github.com/digiPlant/docker-fake-smtp) that will intercept sent emails and save them to workspace, then test suite would read and check the email receiver, title, content.

I wonder if it is worth an effort? Do you have any experience with such tests? Please share your opinion.

Best regards,

Paweł.


SolDevelo
Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

···

Overall cause of these bugs is that Referencedata does not return correct list of users that should get notification (in some cases it caused error, in other it was silently ignored). The problem is that it is hard to catch such errors in unit tests because Referencedata endpoints work properly, logic in service that sends notification (e.g. CCE using Notification) is fine but the calls are wrong. So what we need to test is that we’re sending appropriately to an SMTP server or just correct request to Notification service because there were no bugs related to Notification service itself.

Paweł

On Tue, Apr 24, 2018 at 5:40 PM, josh.zamor@openlmis.org wrote:

Hi Pawel,

Sounds interesting. It’d help to know caused the bugs. As in do we really need to test that we’re sending appropriately to an SMTP server, or do we just need to test the contract of the Notification service?

Best,
Josh

On Tuesday, April 24, 2018 at 8:26:37 AM UTC-7, Paweł Albecki wrote:

Hi all,

On one of our Bug triage meetings, Mary Jo asked about automated end-to-end testing around notifications as we encountered few bugs related to notifications during Regression testing. I promised to raise this topic on Dev Forum.

I’ve also done some research. The simplest way for e2e notification testing would be use our contract tests for this purpose. We could make use of some fake SMPT server (e.g. https://github.com/Nilhcem/FakeSMTP and its docker wrapper https://github.com/digiPlant/docker-fake-smtp) that will intercept sent emails and save them to workspace, then test suite would read and check the email receiver, title, content.

I wonder if it is worth an effort? Do you have any experience with such tests? Please share your opinion.

Best regards,

Paweł.

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/1c42cc72-93e0-4a35-a91e-3917de993e8b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

I’d still advise against adding SMTP into the mix as it still sounds like we do not need to test the SMTP contract, but rather that we needed to test another contract(s). Which one I couldn’t say without knowing the workflows involved. You identified the Reference Data service as the root cause. It seems to me like it a test or set of tests could be written to cover the interaction between:

  1. The service that contacts reference data service

  2. The service that then contacts Notification service

Again I’d only pull in a dummy SMTP tool for tests if we needed to ensure that Notification service is actually properly contacting a SMTP server. That service doesn’t (yet) include any decision logic so it seems a simple set of tests could cover the issue you described. If/when we do get to expanding the Notification service to be multi-channel (email, SMS, in-app, etc), tests that don’t presume that all notifications go out via SMTP would continue to serve our needs in the future.

Obviously I’m commenting a bit in the dark though so if you think I’m missing an important detail please let me know.

Best,

Josh

···

Overall cause of these bugs is that Referencedata does not return correct list of users that should get notification (in some cases it caused error, in other it was silently ignored). The problem is that it is hard to catch such errors in unit tests because Referencedata endpoints work properly, logic in service that sends notification (e.g. CCE using Notification) is fine but the calls are wrong. So what we need to test is that we’re sending appropriately to an SMTP server or just correct request to Notification service because there were no bugs related to Notification service itself.

Paweł

On Tue, Apr 24, 2018 at 5:40 PM, josh.zamor@openlmis.org wrote:

Hi Pawel,

Sounds interesting. It’d help to know caused the bugs. As in do we really need to test that we’re sending appropriately to an SMTP server, or do we just need to test the contract of the Notification service?

Best,

Josh

On Tuesday, April 24, 2018 at 8:26:37 AM UTC-7, Paweł Albecki wrote:

Hi all,

On one of our Bug triage meetings, Mary Jo asked about automated end-to-end testing around notifications as we encountered few bugs related to notifications during Regression testing. I promised to raise this topic on Dev Forum.

I’ve also done some research. The simplest way for e2e notification testing would be use our contract tests for this purpose. We could make use of some fake SMPT server (e.g. https://github.com/Nilhcem/FakeSMTP and its docker wrapper https://github.com/digiPlant/docker-fake-smtp ) that will intercept sent emails and save them to workspace, then test suite would read and check the email receiver, title, content.

I wonder if it is worth an effort? Do you have any experience with such tests? Please share your opinion.

Best regards,

Paweł.

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/1c42cc72-93e0-4a35-a91e-3917de993e8b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Albecki

Software Developer

palbecki@soldevelo.com

I don’t think you are missing anything important. Regarding testing contract between the service and Reference data/Notification, I don’t think we have any good pattern established for this. Impacted code has almost 100% unit test coverage and even if it was 100%, the issue wouldn’t be catched because it’s more like design issue (like in OLMIS-4580, we didn’t expected that supervisory node will be null for some cases) and developer that write unit tests can’t always catch such errors. I think we need some approach for automated testing contract between services in which users/domain experts would be more involved.

Regards,

Paweł


SolDevelo
Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

···

On Tue, Apr 24, 2018 at 6:59 PM, Josh Zamor josh.zamor@villagereach.org wrote:

I’d still advise against adding SMTP into the mix as it still sounds like we do not need to test the SMTP contract, but rather that we needed to test another contract(s). Which one I couldn’t say without knowing the workflows involved. You identified the Reference Data service as the root cause. It seems to me like it a test or set of tests could be written to cover the interaction between:

  1. The service that contacts reference data service
  1. The service that then contacts Notification service

Again I’d only pull in a dummy SMTP tool for tests if we needed to ensure that Notification service is actually properly contacting a SMTP server. That service doesn’t (yet) include any decision logic so it seems a simple set of tests could cover the issue you described. If/when we do get to expanding the Notification service to be multi-channel (email, SMS, in-app, etc), tests that don’t presume that all notifications go out via SMTP would continue to serve our needs in the future.

Obviously I’m commenting a bit in the dark though so if you think I’m missing an important detail please let me know.

Best,

Josh

On Apr 24, 2018, at 9:09 AM, Paweł Albecki palbecki@soldevelo.com wrote:

**

SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com

Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland

Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Overall cause of these bugs is that Referencedata does not return correct list of users that should get notification (in some cases it caused error, in other it was silently ignored). The problem is that it is hard to catch such errors in unit tests because Referencedata endpoints work properly, logic in service that sends notification (e.g. CCE using Notification) is fine but the calls are wrong. So what we need to test is that we’re sending appropriately to an SMTP server or just correct request to Notification service because there were no bugs related to Notification service itself.

Paweł

On Tue, Apr 24, 2018 at 5:40 PM, josh.zamor@openlmis.org wrote:

Hi Pawel,

Sounds interesting. It’d help to know caused the bugs. As in do we really need to test that we’re sending appropriately to an SMTP server, or do we just need to test the contract of the Notification service?

Best,

Josh

On Tuesday, April 24, 2018 at 8:26:37 AM UTC-7, Paweł Albecki wrote:

Hi all,

On one of our Bug triage meetings, Mary Jo asked about automated end-to-end testing around notifications as we encountered few bugs related to notifications during Regression testing. I promised to raise this topic on Dev Forum.

I’ve also done some research. The simplest way for e2e notification testing would be use our contract tests for this purpose. We could make use of some fake SMPT server (e.g. https://github.com/Nilhcem/FakeSMTP and its docker wrapper https://github.com/digiPlant/docker-fake-smtp ) that will intercept sent emails and save them to workspace, then test suite would read and check the email receiver, title, content.

I wonder if it is worth an effort? Do you have any experience with such tests? Please share your opinion.

Best regards,

Paweł.

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/1c42cc72-93e0-4a35-a91e-3917de993e8b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Albecki

Software Developer

palbecki@soldevelo.com

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com

Hi Pawel,

I got delayed. Do we still need a pattern for this? If so could you add the regressions that were found (Jira issues)? I’d like to help if it’s still needed however I think some more context would be useful.

Best,
Josh

···

On Tuesday, April 24, 2018 at 11:24:53 AM UTC-7, Paweł Albecki wrote:

I don’t think you are missing anything important. Regarding testing contract between the service and Reference data/Notification, I don’t think we have any good pattern established for this. Impacted code has almost 100% unit test coverage and even if it was 100%, the issue wouldn’t be catched because it’s more like design issue (like in OLMIS-4580, we didn’t expected that supervisory node will be null for some cases) and developer that write unit tests can’t always catch such errors. I think we need some approach for automated testing contract between services in which users/domain experts would be more involved.

Regards,

Paweł

On Tue, Apr 24, 2018 at 6:59 PM, Josh Zamor josh....@villagereach.org wrote:

I’d still advise against adding SMTP into the mix as it still sounds like we do not need to test the SMTP contract, but rather that we needed to test another contract(s). Which one I couldn’t say without knowing the workflows involved. You identified the Reference Data service as the root cause. It seems to me like it a test or set of tests could be written to cover the interaction between:

  1. The service that contacts reference data service
  1. The service that then contacts Notification service

Again I’d only pull in a dummy SMTP tool for tests if we needed to ensure that Notification service is actually properly contacting a SMTP server. That service doesn’t (yet) include any decision logic so it seems a simple set of tests could cover the issue you described. If/when we do get to expanding the Notification service to be multi-channel (email, SMS, in-app, etc), tests that don’t presume that all notifications go out via SMTP would continue to serve our needs in the future.

Obviously I’m commenting a bit in the dark though so if you think I’m missing an important detail please let me know.

Best,

Josh

On Apr 24, 2018, at 9:09 AM, Paweł Albecki palb...@soldevelo.com wrote:

Overall cause of these bugs is that Referencedata does not return correct list of users that should get notification (in some cases it caused error, in other it was silently ignored). The problem is that it is hard to catch such errors in unit tests because Referencedata endpoints work properly, logic in service that sends notification (e.g. CCE using Notification) is fine but the calls are wrong. So what we need to test is that we’re sending appropriately to an SMTP server or just correct request to Notification service because there were no bugs related to Notification service itself.

Paweł

On Tue, Apr 24, 2018 at 5:40 PM, josh....@openlmis.org wrote:

Hi Pawel,

Sounds interesting. It’d help to know caused the bugs. As in do we really need to test that we’re sending appropriately to an SMTP server, or do we just need to test the contract of the Notification service?

Best,

Josh

On Tuesday, April 24, 2018 at 8:26:37 AM UTC-7, Paweł Albecki wrote:

Hi all,

On one of our Bug triage meetings, Mary Jo asked about automated end-to-end testing around notifications as we encountered few bugs related to notifications during Regression testing. I promised to raise this topic on Dev Forum.

I’ve also done some research. The simplest way for e2e notification testing would be use our contract tests for this purpose. We could make use of some fake SMPT server (e.g. https://github.com/Nilhcem/FakeSMTP and its docker wrapper https://github.com/digiPlant/docker-fake-smtp ) that will intercept sent emails and save them to workspace, then test suite would read and check the email receiver, title, content.

I wonder if it is worth an effort? Do you have any experience with such tests? Please share your opinion.

Best regards,

Paweł.

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.

To post to this group, send email to openlm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/1c42cc72-93e0-4a35-a91e-3917de993e8b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Albecki

Software Developer

palb...@soldevelo.com

**

SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com

Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland

Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Paweł Albecki

    Software Developer

     palb...@soldevelo.com


SolDevelo
Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

The Issues I was talking about:
https://openlmis.atlassian.net/browse/OLMIS-4552

https://openlmis.atlassian.net/browse/OLMIS-4580

https://openlmis.atlassian.net/browse/OLMIS-4596

https://openlmis.atlassian.net/browse/OLMIS-4595


SolDevelo
Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

···

On Wed, May 2, 2018 at 5:11 PM, josh.zamor@openlmis.org wrote:

Hi Pawel,

I got delayed. Do we still need a pattern for this? If so could you add the regressions that were found (Jira issues)? I’d like to help if it’s still needed however I think some more context would be useful.

Best,
Josh

On Tuesday, April 24, 2018 at 11:24:53 AM UTC-7, Paweł Albecki wrote:

I don’t think you are missing anything important. Regarding testing contract between the service and Reference data/Notification, I don’t think we have any good pattern established for this. Impacted code has almost 100% unit test coverage and even if it was 100%, the issue wouldn’t be catched because it’s more like design issue (like in OLMIS-4580, we didn’t expected that supervisory node will be null for some cases) and developer that write unit tests can’t always catch such errors. I think we need some approach for automated testing contract between services in which users/domain experts would be more involved.

Regards,

Paweł

On Tue, Apr 24, 2018 at 6:59 PM, Josh Zamor josh....@villagereach.org wrote:

I’d still advise against adding SMTP into the mix as it still sounds like we do not need to test the SMTP contract, but rather that we needed to test another contract(s). Which one I couldn’t say without knowing the workflows involved. You identified the Reference Data service as the root cause. It seems to me like it a test or set of tests could be written to cover the interaction between:

  1. The service that contacts reference data service
  1. The service that then contacts Notification service

Again I’d only pull in a dummy SMTP tool for tests if we needed to ensure that Notification service is actually properly contacting a SMTP server. That service doesn’t (yet) include any decision logic so it seems a simple set of tests could cover the issue you described. If/when we do get to expanding the Notification service to be multi-channel (email, SMS, in-app, etc), tests that don’t presume that all notifications go out via SMTP would continue to serve our needs in the future.

Obviously I’m commenting a bit in the dark though so if you think I’m missing an important detail please let me know.

Best,

Josh

On Apr 24, 2018, at 9:09 AM, Paweł Albecki palb...@soldevelo.com wrote:

Overall cause of these bugs is that Referencedata does not return correct list of users that should get notification (in some cases it caused error, in other it was silently ignored). The problem is that it is hard to catch such errors in unit tests because Referencedata endpoints work properly, logic in service that sends notification (e.g. CCE using Notification) is fine but the calls are wrong. So what we need to test is that we’re sending appropriately to an SMTP server or just correct request to Notification service because there were no bugs related to Notification service itself.

Paweł

On Tue, Apr 24, 2018 at 5:40 PM, josh....@openlmis.org wrote:

Hi all,

On one of our Bug triage meetings, Mary Jo asked about automated end-to-end testing around notifications as we encountered few bugs related to notifications during Regression testing. I promised to raise this topic on Dev Forum.

I’ve also done some research. The simplest way for e2e notification testing would be use our contract tests for this purpose. We could make use of some fake SMPT server (e.g. https://github.com/Nilhcem/FakeSMTP and its docker wrapper https://github.com/digiPlant/docker-fake-smtp ) that will intercept sent emails and save them to workspace, then test suite would read and check the email receiver, title, content.

I wonder if it is worth an effort? Do you have any experience with such tests? Please share your opinion.

Best regards,

Paweł.

Hi Pawel,

Sounds interesting. It’d help to know caused the bugs. As in do we really need to test that we’re sending appropriately to an SMTP server, or do we just need to test the contract of the Notification service?

Best,

Josh

On Tuesday, April 24, 2018 at 8:26:37 AM UTC-7, Paweł Albecki wrote:

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.

To post to this group, send email to openlm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/1c42cc72-93e0-4a35-a91e-3917de993e8b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Albecki

Software Developer

palb...@soldevelo.com

**

SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com

Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland

Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Paweł Albecki

    Software Developer

     palb...@soldevelo.com


SolDevelo
Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/682e466e-ae84-40a6-82a1-63d8c75bed9d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Albecki

    Software Developer

     palbecki@soldevelo.com