The way to improve searching for existing test cases and automated tests


#1

Hi all,
for the last few days, I have been working on OLMIS-5550 in order to find a workflow/pattern which will allow us to easily track which features and cases are already covered by tests in order to avoid duplications, and those that are not covered yet. I also need to document this pattern in an appropriate place. I have come up with the solution below so far, and I would appreciate your feedback very much.

I think that we need to create and then, add suitable labels which will serve as keywords concerning a given test case’s content to all test cases. For instance, e.g. OLMIS-4259 could be labelled with the following labels: “Orders”, “StockCards”, “Fulfillment” and “LocalFulfillment”. I browsed through all test cases with the “To Do” and “Roadmap” statuses in our system, and came up with the following labels (some of them already exist on our Jira):

  1. Adjustments
  2. Administration
  3. Alerts
  4. Automated
  5. Autosaving
  6. Calculations
  7. Catalog
  8. CCE
  9. Comments
  10. Configuration
  11. CT (i.e. Contract Test)
  12. DemoData
  13. Email
  14. EmergencyRequisitions
  15. Equipment
  16. ExternalFulfillment
  17. Facilities
  18. FacilityTypes
  19. FHIR
  20. Filtering
  21. FT (i.e. Functional Test)
  22. FTP
  23. Fulfillment
  24. FTAPs
  25. GeographicZones
  26. ISAs
  27. LocalFulfillment
  28. Login
  29. Notifications
  30. Offline
  31. Orders
  32. Password
  33. Permissons
  34. PoDs
  35. PhysicalInventory
  36. ProcessingPeriods
  37. ProcessingSchedules
  38. Products
  39. Profile
  40. Programs
  41. Reasons
  42. Receipts
  43. RegularRequisitions
  44. Reporting
  45. Requisitions
  46. RequisitionGroups
  47. Roles
  48. Searching
  49. ServiceAccounts
  50. Shipment
  51. Sorting
  52. StockCards
  53. StockManagement
  54. StockOnHand
  55. SupervisoryNodes
  56. SupplyLines
  57. Tags
  58. Templates
  59. UI
  60. Users
  61. VaccineSBR
  62. VVM
  63. Workflow

I think that using such labels will make searching for suitable test cases much faster. If you agree with this idea, I propose that we create the following follow-up tickets:

  • The ticket to browse through existing contract and functional tests and then, browsing through the manual test cases and labeling them with the “Automated” and “CT”/“FT” label, depending on the type of automated tests - In my view, we need to have an easy way to check what features are covered by automated tests and I think this would be the most effective one to achieve this. Also, during my research, I learned that it is done thus in some projects;
  • The ticket to create a new section in the “Test Strategy” document concerning the labels and describing how to use them with the test cases. Alternatively, a separate Confluence page could be created;
  • The ticket to browse through all manual test cases in the system and label them with the new labels.
    Regards,

Joanna


(Wes Brown) #2

Thanks for this, Joanna!

I like the idea of organizing our test cases but it seems like you are mixing different concepts into the same field and I wonder if that will make this confusing both to categorize and to search. For example, the list contains functional areas (StockManagement, Requisitions), actions (Sorting, Searching), state (Offline), and more broad categories (Configuration, UI). Also, given the length of this list, it seems unlikely that people will remember to only use these items which will likely result in a many other labels being created that only really make sense to the person doing the labeling.

Based on that, I have a couple of suggestions:

  1. Can we create a more simplified list with fewer, more broad categories?
  2. Can we use the existing Component field in JIRA to track the functional area rather than duplicate this information as a label?
  3. Can we create a custom field for this test case categorization rather than reuse the Label field?
    Thanks,

-Wes

···

On Friday, October 19, 2018 at 2:56:38 AM UTC-4, jbebak@soldevelo.com wrote:

Hi all,
for the last few days, I have been working on OLMIS-5550 in order to find a workflow/pattern which will allow us to easily track which features and cases are already covered by tests in order to avoid duplications, and those that are not covered yet. I also need to document this pattern in an appropriate place. I have come up with the solution below so far, and I would appreciate your feedback very much.

I think that we need to create and then, add suitable labels which will serve as keywords concerning a given test case’s content to all test cases. For instance, e.g. OLMIS-4259 could be labelled with the following labels: “Orders”, “StockCards”, “Fulfillment” and “LocalFulfillment”. I browsed through all test cases with the “To Do” and “Roadmap” statuses in our system, and came up with the following labels (some of them already exist on our Jira):

  1. Adjustments
  2. Administration
  3. Alerts
  4. Automated
  5. Autosaving
  6. Calculations
  7. Catalog
  8. CCE
  9. Comments
  10. Configuration
  11. CT (i.e. Contract Test)
  12. DemoData
  13. Email
  14. EmergencyRequisitions
  15. Equipment
  16. ExternalFulfillment
  17. Facilities
  18. FacilityTypes
  19. FHIR
  20. Filtering
  21. FT (i.e. Functional Test)
  22. FTP
  23. Fulfillment
  24. FTAPs
  25. GeographicZones
  26. ISAs
  27. LocalFulfillment
  28. Login
  29. Notifications
  30. Offline
  31. Orders
  32. Password
  33. Permissons
  34. PoDs
  35. PhysicalInventory
  36. ProcessingPeriods
  37. ProcessingSchedules
  38. Products
  39. Profile
  40. Programs
  41. Reasons
  42. Receipts
  43. RegularRequisitions
  44. Reporting
  45. Requisitions
  46. RequisitionGroups
  47. Roles
  48. Searching
  49. ServiceAccounts
  50. Shipment
  51. Sorting
  52. StockCards
  53. StockManagement
  54. StockOnHand
  55. SupervisoryNodes
  56. SupplyLines
  57. Tags
  58. Templates
  59. UI
  60. Users
  61. VaccineSBR
  62. VVM
  63. Workflow

I think that using such labels will make searching for suitable test cases much faster. If you agree with this idea, I propose that we create the following follow-up tickets:

  • The ticket to browse through existing contract and functional tests and then, browsing through the manual test cases and labeling them with the “Automated” and “CT”/“FT” label, depending on the type of automated tests - In my view, we need to have an easy way to check what features are covered by automated tests and I think this would be the most effective one to achieve this. Also, during my research, I learned that it is done thus in some projects;
  • The ticket to create a new section in the “Test Strategy” document concerning the labels and describing how to use them with the test cases. Alternatively, a separate Confluence page could be created;
  • The ticket to browse through all manual test cases in the system and label them with the new labels.
    Regards,

Joanna


(Sebastian Brudziński) #3

In addition to what Wes has stated, I don’t think it makes sense to have labels/categories for our services - requisition, stockmanagement, etc. We are already using the component field for this, so it seems like an unwanted duplication.

I like the idea of having a predefined list of categories/groups rather than using labels too. That should make both searching and adding them to new test cases easier (Jira should be able to generate that field as a dropdown then)

I don’t exactly get what you are referring to when you say we should label contract/functional tests. The goal is to drop manual test case as soon as we fully cover given case with an automated test. We also won’t have a related ticket with each functional/contract test. Can you clarify this a bit?

I’m mostly interested in whether after those changes you will be able to easily find a test case or say with confidence that a case is not covered for any given feature in a short time (like less than 5 minutes). Do you think this categorization is enough to achieve that?

Thanks for writing this down!

Best regards,

Sebastian.

···

Sebastian Brudziński
Technical Leader
sbrudzinski@soldevelo.com


#4

Hi Wes and Sebastian,
thank you both for your suggestions!

I managed to decrease the number of the proposed key words - I excluded those that overlap with the “Component” and “Label” fields, and those that are too specific. Instead, I left only those that pertain to specific screens or features that will make it easier to distinguish the test cases from each other. I think the categories can’t be broader, as those under the “Component” and “Label” fields are broad, and they don’t render searching for the test cases fast enough - they are not specific enough and there are too few of them. Though of course I think we should still keep using them, along with the new custom field. I think the latter should be called “Keyword”, and it should be a drop-down list from which one could select suitable options, as Sebastian suggested, so that we don’t create countless options, as is the case as far as the “Label” field is concerned.

As for the “Automated”, “CT” and “FT” keywords, I thought that we should label the tickets with manual test cases with these labels/keywords, so that we know what features that had been previously covered with manual tests are now covered with functional/contract tests - it’s easier to track it thus then with the use of the links on Jira between the ticket with a manual test case and e.g. the one concerning the creation of a contract/functional test. This could be helpful in making next steps in test automation, e.g. to determine the priorities - for instance, to tell which functional/contract test should be created first. Also, probably this could help the person who would be creating such an automated test - they could check whether there is e.g. an automated test for a similar feature created already, and look at the steps contained in the manual test case, which also could be some tip on how to write the automated test. But if you think that marking the manual test cases with the labels/keywords concerning automated tests is not necessary, I will exclude those keywords.

All in all, in my view, adding such a custom field to our Jira would quicken looking for suitable manual test cases greatly - I think that after introducing the field and marking the test cases, searching for manual test cases concerning a given feature would take up to 5 minutes, even for a person who is not familiar with our test cases.

The current list of keywords is the following:

  1. Adjustments
  2. Alerts
  3. Automated
  4. Calculations
  5. Catalog
  6. CT
  7. Email
  8. EmergencyRequisitions
  9. Facilities
  10. FacilityTypes
  11. FT
  12. GeographicZones
  13. ISAs
  14. Login
  15. Password
  16. Permissons
  17. PoDs
  18. PhysicalInventory
  19. ProcessingPeriods
  20. ProcessingSchedules
  21. Products
  22. Profile
  23. Programs
  24. Reasons
  25. Receipts
  26. RegularRequisitions
  27. RequisitionGroups
  28. Roles
  29. ServiceAccounts
  30. Shipment
  31. StockCards
  32. StockOnHand
  33. SupervisoryNodes
  34. SupplyLines
  35. Tags
  36. Templates
  37. Users
  38. VVM
···

On Friday, October 19, 2018 at 6:05:19 PM UTC+2, Sebastian Brudziński wrote:

In addition to what Wes has stated, I don’t think it makes sense to have labels/categories for our services - requisition, stockmanagement, etc. We are already using the component field for this, so it seems like an unwanted duplication.

I like the idea of having a predefined list of categories/groups rather than using labels too. That should make both searching and adding them to new test cases easier (Jira should be able to generate that field as a dropdown then)

I don’t exactly get what you are referring to when you say we should label contract/functional tests. The goal is to drop manual test case as soon as we fully cover given case with an automated test. We also won’t have a related ticket with each functional/contract test. Can you clarify this a bit?

I’m mostly interested in whether after those changes you will be able to easily find a test case or say with confidence that a case is not covered for any given feature in a short time (like less than 5 minutes). Do you think this categorization is enough to achieve that?

Thanks for writing this down!

Best regards,

Sebastian.

On Fri, Oct 19, 2018 at 3:08 PM wes....@villagereach.org wrote:

Thanks for this, Joanna!

I like the idea of organizing our test cases but it seems like you are mixing different concepts into the same field and I wonder if that will make this confusing both to categorize and to search. For example, the list contains functional areas (StockManagement, Requisitions), actions (Sorting, Searching), state (Offline), and more broad categories (Configuration, UI). Also, given the length of this list, it seems unlikely that people will remember to only use these items which will likely result in a many other labels being created that only really make sense to the person doing the labeling.

Based on that, I have a couple of suggestions:

  1. Can we create a more simplified list with fewer, more broad categories?
  2. Can we use the existing Component field in JIRA to track the functional area rather than duplicate this information as a label?
  3. Can we create a custom field for this test case categorization rather than reuse the Label field?
    Thanks,

-Wes

On Friday, October 19, 2018 at 2:56:38 AM UTC-4, jbe...@soldevelo.com wrote:

Hi all,
for the last few days, I have been working on OLMIS-5550 in order to find a workflow/pattern which will allow us to easily track which features and cases are already covered by tests in order to avoid duplications, and those that are not covered yet. I also need to document this pattern in an appropriate place. I have come up with the solution below so far, and I would appreciate your feedback very much.

I think that we need to create and then, add suitable labels which will serve as keywords concerning a given test case’s content to all test cases. For instance, e.g. OLMIS-4259 could be labelled with the following labels: “Orders”, “StockCards”, “Fulfillment” and “LocalFulfillment”. I browsed through all test cases with the “To Do” and “Roadmap” statuses in our system, and came up with the following labels (some of them already exist on our Jira):

  1. Adjustments
  2. Administration
  3. Alerts
  4. Automated
  5. Autosaving
  6. Calculations
  7. Catalog
  8. CCE
  9. Comments
  10. Configuration
  11. CT (i.e. Contract Test)
  12. DemoData
  13. Email
  14. EmergencyRequisitions
  15. Equipment
  16. ExternalFulfillment
  17. Facilities
  18. FacilityTypes
  19. FHIR
  20. Filtering
  21. FT (i.e. Functional Test)
  22. FTP
  23. Fulfillment
  24. FTAPs
  25. GeographicZones
  26. ISAs
  27. LocalFulfillment
  28. Login
  29. Notifications
  30. Offline
  31. Orders
  32. Password
  33. Permissons
  34. PoDs
  35. PhysicalInventory
  36. ProcessingPeriods
  37. ProcessingSchedules
  38. Products
  39. Profile
  40. Programs
  41. Reasons
  42. Receipts
  43. RegularRequisitions
  44. Reporting
  45. Requisitions
  46. RequisitionGroups
  47. Roles
  48. Searching
  49. ServiceAccounts
  50. Shipment
  51. Sorting
  52. StockCards
  53. StockManagement
  54. StockOnHand
  55. SupervisoryNodes
  56. SupplyLines
  57. Tags
  58. Templates
  59. UI
  60. Users
  61. VaccineSBR
  62. VVM
  63. Workflow

I think that using such labels will make searching for suitable test cases much faster. If you agree with this idea, I propose that we create the following follow-up tickets:

  • The ticket to browse through existing contract and functional tests and then, browsing through the manual test cases and labeling them with the “Automated” and “CT”/“FT” label, depending on the type of automated tests - In my view, we need to have an easy way to check what features are covered by automated tests and I think this would be the most effective one to achieve this. Also, during my research, I learned that it is done thus in some projects;
  • The ticket to create a new section in the “Test Strategy” document concerning the labels and describing how to use them with the test cases. Alternatively, a separate Confluence page could be created;
  • The ticket to browse through all manual test cases in the system and label them with the new labels.
    Regards,

Joanna

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.

To post to this group, send email to openlm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/7b723f6e-bd58-4c21-8247-8fd5e8a2bcac%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


Sebastian Brudziński
Technical Leader
sbrud...@soldevelo.com


SolDevelo
Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41


(Klaudia Pałkowska) #5

Hi all,

I like the idea of using a new “Keyword” field. It would be intuitive and clear, even for a person who doesn’t know anything about our test cases pattern. As for the “Automated” keyword, in my opinion, we don’t need it. A test case should be marked as Dead whether it’s covered by automated tests. For “CT” and “FT”, I think it could be applicable to marking whether the previously uncovered case is currently covered by an automated test but only if a few of them are still needed. Otherwise, we should just move it to Dead. Additionally, since we decided to leave UI specific testing and the fact that manual test cases check UI (not API) is obvious, I would say the “UI” keyword is also unnecessary.

Also, I suggest adding one for stock-based requisitions, i.e. StockBased, SBR or simply StockBasedRequisitions. What do you think?

Kindest regards,
Klaudia


(Sebastian Brudziński) #6

The reason I don’t see a point in adding Automated/CT/FT keywords is that as soon as the test case is fully automated, we want to mark that test case as dead and stop maintaining it anymore. How exactly we determine when a test case is fully automated and who approves that is still pending discussion, but the goal is definitely to reduce the number of manual tests we need to manage.

Overall, I think the approach with predefined test-related keywords sounds OK. I still have minor suggestions/feedback on concrete keywords on that list, but we can get to that when we start introducing them.

Best regards,
Sebastian.


#7

Hi all,
thank you for your suggestions.
OK, if you don’t see the point in marking the automated test cases with the Automated/CT/FT keywords, I’m alright with it - you convinced me. I think that the “SBR” keyword is not necessary because we have a similar label, and I think that we shouldn’t have label-keyword duplicates. Moreover, the “UI” keyword doesn’t appear on the newer list of proposed keywords because I also found it unnecessary after all. Though I guess that we’ll decide on the specific keywords that need to be created during their implementation.
Just to be sure - can I consider my part of the OLMIS-5550 ticket as done? I’m asking because I suppose that we have reached some conclusion.
Best regards,
Joanna


(Sebastian Brudziński) #8

Hi,

have we created the tickets for that follow up work yet? Once they are created and added to https://openlmis.atlassian.net/browse/OLMIS-5651 we can consider that part finished.

Best,
Sebastian.


#9

Hi Sebastian,
no, I haven’t created the follow-up tickets yet but I will do so and add them to the epic, as you suggested.
Best,
Joanna