I have written a CI/CD analysis and improvement plan.
I will post the content below for convenience of reading, but refer to this link for latest version: https://docs.google.com/document/d/1TZ55h0F1fHr901bNN76-A5cc_7PeiD02rla5F9eyPEk/
since the content may change in the future.
OpenLMIS 3.0 has 10 git repos as listed below:
Needs CI job
Needed in CI pipeline
Needed in CD
This repo is meant to demonstrate how to create a new service for OpenLMIS 3.0. It’s not meant to be used in prod env.
This is a starter kit for devs to start a new service
This is a base docker image intended mainly for devs to use locally
As we can see in the table, the first three will be the main players in CI/CD.
Majority of future new repos will probably be added in the service category. They can also follow the plan described below to fit in the CI/CD picture.
Most jobs are triggered by git push individually.
Tests, Sonar and ERD generation are ran for some, but not all the services.
Currently there is no pipeline.
Types of Jenkins jobs
An improved CI will consist of the following parts, each with its own objectives and provides different aspects of visualization.
Each individual job is focused to verify behavioral correctness, code quality of one service, build and publish artifacts as needed.
Behavioral correctness will be verified by unit tests, integration tests, component tests, etc. Code quality will be verified by PMD, CheckStyle, FindBugs and Sonar, etc.
Published artifacts will include: test reports, test coverage reports, API docs, ERD, docker images, etc.
Each individual job should be self-contained and self-sufficient, meaning that it needs no other services to be running to perform its own tasks and its own fail/success status should be enough for the team to know how the latest code of a certain service is doing.
Contract test jobs
Contract tests should be organized around business scenarios that requires multiple services to work together.
Business analyst, developers and QAs work together to determine which scenarios to write contract tests for.
The automated tests should be written by developers while they are developing those features that need multiple services to work together.
Contract tests should be sending real http requests to and verifying real http response from running services. Mocking and stubbing techniques should reside within component tests of each single service.
End to end test jobs
Since OpenLMIS 3.0 has no UI at the moment, E2E tests and contract tests will be indistinguishable from a technical perspective.
But they are different in terms of business scenario.
Contract tests could often emerge from individual stories, while E2E tests would more commonly revolve around user journeys.
Which means E2E tests are concerned with: for a certain typical user, who has a series of tasks to perform, will the current system as whole be successful in helping that user on that journey.
In the future, when UI is developed, automated tests that drives through the UI could also be a part of E2E test.
As the name suggests, these jobs will be responsible for deploying individual services or the OpenLMIS system as a whole.
Pipelines are ways to organize Jenkins jobs in trigger order and logical grouping.
For a monolithic web app, its pipeline will typically look like the following:
Since OpenLMIS is using a micro services architecture, which enables each service to be developed and deployed separately, that also makes its pipelines more complex.
Each service could have its own pipeline, which is responsible for running tests of various kinds, publishing artifacts, deploying to different environments.
Monitors provide an visualized overview of the status at different levels.
For example, there could be: a monitor that shows latest test results of different services, a monitor that shows latest deployment status to UAT environment, etc.
It could also be displayed on a large TV located near the development team to provide constant visual feedback of how things are going.
Individual jobs should be triggered by each git push.
Contract and E2E test jobs should ideally be triggered if any code in any git repo is changed. As the number of tests increase and time consumption of those jobs go up, they can be adjusted to run for each time period(once each hour, 3 times a day, etc).
Deployment job of a certain service should be triggered after all other steps in its pipeline finishes successfully.
Deployment job of the whole system should be triggered manually, whenever the we see fit.
In addition to that, each job could be triggered manually, to enable flexibility.
Impact/challenge for the team
Writing different kinds of automated tests ;
Identifying, distinguishing scenarios for contract and E2E tests;
Maintaining a healthy pipeline;
Those things will require continuous effort from the team to form habits and cultivate skills.
Kick start contract and E2E tests 1-2 weeks
Creating new jobs, publishing correct artifacts 1 week
Configure deployment environments, create deployment scripts and jobs 1-2 week
Organize jobs, pipelines and monitors 1 week