Challenges related to setting up Reporting Stack

Paulina_Buzderewicz · May 13, 2020, 11:41am

Hello,

According to our yesterday’s call concerning challenges of setting up the Reporting Stack, I would like to start the topic for listing all of the potential issues that may come up during working on it.

For now we are aware of the following challenges:

Some variables (such as URLs) may be hard-coded in reporting configuration. It also applies to the OpenLMIS UI. All these places should be recognized and listed in this post along with working on Reporting Stack set-up.
Load balancer configuration - it is important that the ports match the settings in the repositories.
Building Superset - in a case when docker didn’t use cache different issues with dependencies appeared every build.
Configuration of nifi flows needs adjustments for each instance.

The dedicated wiki page can be found under this link: https://openlmis.atlassian.net/wiki/spaces/COV/pages/867992085/Setting+up+Reporting+Stack.

If there are any mentioned or new issues met during setting up the Reporting Stack they should be described in this forum post.

Best regards,
Paulina.

joshzamor · May 13, 2020, 4:36pm

Thanks for starting this @Paulina_Buzderewicz.

I’d love to see this list, and I’d also encourage us to think about some of the advancements that have been taking place in NiFi since we last updated. It looks like NiFi 1.10 introduced parameters as well as a new stateless runtime. On the surface this sounds like it could help some of the pain points I remember from our NiFi flows from the last year.

ohinc · May 18, 2020, 9:31am

Hi all,

As we’ve just run into the dependency issues, I’d like to point out what’s the current approach of handling it.

An example dependency issue looks like this:

ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-mt11c1gs/markupsafe/setup.py'"'"'; __file__='"'"'/tmp/pip-install-mt11c1gs/markupsafe/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-bqww1qrl
         cwd: /tmp/pip-install-mt11c1gs/markupsafe/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-mt11c1gs/markupsafe/setup.py", line 6, in <module>
        from setuptools import setup, Extension, Feature
    ImportError: cannot import name 'Feature'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Service 'superset' failed to build: The command '/bin/sh -c useradd -U -m superset &&     mkdir /etc/superset  &&     mkdir /var/lib/superset &&     chown -R superset:superset /etc/superset &&     chown -R superset:superset /var/lib/superset &&     apt-get update &&     apt-get install -y         rsync         build-essential         curl         default-libmysqlclient-dev         freetds-bin         freetds-dev         postgresql-client         libffi-dev         libldap2-dev         libpq-dev         libsasl2-dev         libssl1.0 &&     apt-get clean &&     rm -r /var/lib/apt/lists/* &&     pip install --no-cache-dir -r requirements.txt &&     rm requirements.txt &&     pip install --no-cache-dir         flask-cors==3.0.3         flask-mail==0.9.1         flask-JWT-Extended==3.18.2         flask-oauth==0.12         flask_oauthlib==0.9.5         requests-oauthlib==1.1.0         gevent==1.2.2         impyla==0.14.0         infi.clickhouse-orm==1.0.2         psycopg2-binary==2.8.3         pyathena==1.2.5         pybigquery==0.4.10         pyhive==0.5.1         pyldap==2.4.28         redis==2.10.5         sqlalchemy-clickhouse==0.1.5.post0         sqlalchemy-redshift==0.7.1         werkzeug==0.14.1 &&         pip install --no-cache-dir git+https://github.com/OpenLMIS/superset-patchup.git &&     pip install superset==0.29.0rc7 &&     pip uninstall -y sqlalchemy pandas &&     pip install sqlalchemy==1.2.18 pandas==0.23.4' returned a non-zero code: 1

And it was thrown while executing pip install --no-cache-dir -r requirements.txt. As seen in the error above - there was an issue with the markupsafe library. The best way to update the required versions of dependencies is to do it by automatically updating the requirements.txt file:

Download proper superset-patchup repository (e. g. https://github.com/OpenLMIS/superset-patchup/ which is used for core)
Install it locally - pip3.6 install git+https://github.com/onaio/superset-patchup.git<@optional version>. Please note that:
- sudo might be required
- pip3.6 was used to indicate Python in version 3.6 is required
- If superset_patchup is not recognized in file setup.py it can be temporarily set to a fixed value there, e. g. version='0.1.7'
Use pip-compile --rebuild --output-file requirements.txt setup.py to regenerate requirements file
Update the requirements file in the distro repository

Best,
Oskar

ohinc · May 18, 2020, 12:53pm

Hi all,

Another issue we’ve encountered was related to the AWS Load Balanced timeout that was by default set to 60 seconds. Changing the ‘Idle timeout’ property to 900 seconds resolved it. In the logs, the error was presented as below:

Traceback (most recent call last):
File “http/client.py”, line 546, in _get_chunk_left
File “http/client.py”, line 513, in _read_next_chunk_size
ValueError: invalid literal for int() with base 16: b’’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “http/client.py”, line 578, in _readinto_chunked
File “http/client.py”, line 548, in _get_chunk_left
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “site-packages/urllib3/response.py”, line 302, in _error_catcher
File “site-packages/urllib3/response.py”, line 384, in read
File “http/client.py”, line 449, in read
File “http/client.py”, line 483, in readinto
File “http/client.py”, line 594, in _readinto_chunked
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “bin/docker-compose”, line 6, in
File “compose/cli/main.py”, line 71, in main
File “compose/cli/main.py”, line 127, in perform_command
File “compose/cli/main.py”, line 282, in build
File “compose/project.py”, line 373, in build
File “compose/service.py”, line 1051, in build
File “compose/progress_stream.py”, line 26, in stream_output
File “compose/utils.py”, line 61, in split_buffer
File “compose/utils.py”, line 37, in stream_as_text
File “site-packages/docker/api/client.py”, line 307, in _stream_helper
File “site-packages/urllib3/response.py”, line 401, in read
File “contextlib.py”, line 99, in exit
File “site-packages/urllib3/response.py”, line 320, in _error_catcher
urllib3.exceptions.ProtocolError: (‘Connection broken: IncompleteRead(0 bytes read)’, IncompleteRead(0 bytes read))

Best,
Oskar

aswiszcz · May 22, 2020, 12:46pm

Hello,

Concerning to list of hardcoded values we still come across new ones. At the moment, the list of these values is as follows:

Generate Measures:

getAccessToken :
${baseurl}/oauth/token?grant_type=password&username=${username}&password=password
to
${baseUrl}/api/oauth/token?grant_type=password&username=${username}&password=${password}
Check Measure > Invoke http:
${base_url}/api/oauth/token?grant_type=client_credentials
to
${baseUrl}/api/oauth/token?grant_type=client_credentials
and
${base_url}/hapifhir/Measure
to
${baseUrl}/hapifhir/Measure
Generate Measures > Invoke HTTP
https://uat.openlmis.org/hapifhir/Measure
to
${baseUrl}/hapifhir/Measure
Generate Measures > Check Measure > GenerateFlowFile:
https://uat.openlmis.org](https://uat.openlmis.org/
to
${baseUrl}
Generate Measures > Generate list of facility type approved products > get products > Generate Flow File
https://uat.openlmis.org/api
to
${baseUrl}
${baseurl}/facilityTypeApprovedProducts?access_token=${access_token}&facilityType=health_center
to
${baseUrl}/facilityTypeApprovedProducts?access_token=${access_token}

Generate Measure Reports:

Get access token : ${base_url}/oauth/token?grant_type=password&username=${username}&password=${password}
to
${baseUrl}/oauth/token?grant_type=password&username=${username}&password=${password}
GET requisitions from /api: ${base_url}/api/requisitions/search?access_token=${access_token}
to
${baseUrl}/api/requisitions/search?access_token=${access_token}
GET requisition line ites from /api: ${base_url}/api/requisitions/${requisitionId}?access_token=${access_token}
to
${baseUrl}/api/requisitions/${requisitionId}?access_token=${access_token}
Get Measures & Locations > GenerateFlowFile username: administrator
to
${username}
Get Measures & Locations > Get Measures from FHIR server > Invoke HTTP:
${base_url}/api/oauth/token?grant_type=client_credentials
to
${baseUrl}/api/oauth/token?grant_type=client_credentials
and
${base_url}/hapifhir/Measure
to
${baseUrl}/hapifhir/Measure
Get Measures & Locations > Get Locations from FHIR server > Invoke HTTP:
${base_url}/hapifhir/Location
to
${baseUrl}/hapifhir/Location
generate measure reports > Invoke HTTP:
${base_url}/hapifhir/MeasureReport
to
${baseUrl}/hapifhir/MeasureReport

In case of encountering further hardcoded values the list will be updated with new ones, of course each encountered value is converted by us into a variable from configuration.

Best regards,
Adrian

Paulina_Buzderewicz · May 25, 2020, 9:50am

Hi,

The flows for requisition, referencedata, superset permissions and materialized views work properly for covid reference instance. However there are two additional flows: Generate Measure Reports and Generate Measures. Actually we are not sure, what are these flows responsible for. They are not used in our Superset reports. At the end of each flow they send POST requests with Measure Resources to /hapifhir/Measure and /hapifhir/MeasureReport endpoints.

We are asking about it, because one of these flows - Generate Measures - does not work properly. During the running flow, we get an error about JSON content being not valid. There is one place in the Flow where JSON value is replaced to an empty string, which may cause this exception, but we are not sure here. You can see the error on the screen below:

Does anyone here understand this flow and knows whether we should include it in our Covid project? It also looks like the flow doesn’t work in Core and Malawi. Moreover, other implementations (Angola, SELV) do not use them at all. This is the only thing (apart from configuring SSL) which is stopping us from considering work on setting up the Reporting stack as completed.

Best regards,
Paulina.

joshzamor · May 26, 2020, 11:50pm

Hi @Paulina_Buzderewicz,

They aren’t used in Superset, rather that data is intended for OpenHIE’s HMIS (e.g. DHIS2) aggregate supply chain reporting. The flow delivers it to the FHIR server (hapifhir) for later consumption/transmission.

Yes, we should likely have this working as in the COVID project we believe one/atleast one of the implementations is interested in sending these reports to DHIS2. I’ll know more later this week on that implementation and timing.

The flow should work - we’re going to need to fix it in OpenLMIS at the very least. It also is in use in the Malawi implementation, to the best of my knowledge. It feeds their HIE system’s format, which should eventually make it to their DHIS2 instance. If it really isn’t working we should be investigating that with the Malawi team - it’s the first I’ve heard about a possible issue. I believe also we’ll see adoption of this for Angola take place relatively soon as well - though again to the best of my knowledge that’s coming and not expected to be in place yet.

ohinc · May 27, 2020, 3:21pm

Hi all,

Recently we’ve prepared a step-by-step instruction that explains step-by-step what actions should be performed to create and configure a completely new instance from scratch - you can find it here.

Best,
Oskar

Paulina_Buzderewicz · May 28, 2020, 1:08pm

Hi all,

I would like to describe yet another issue related to the reporting stack. While creating reporting stack for Covid we took XML files from the core’s repository. Later we encountered some weird behavior, like Basic Authentication Username and Password properties not being automatically set in Generate Measures and Generate Measure Reports flows after each redeploy. It turned out that the preload script was adjusted to different templates. It means that the XML files for those two flows are outdated in core’s ref-distro. I replaced them with the proper ones for the covid project.

We’re still not sure if the flows work correctly. We are concerned about the flows, because we don’t see data transfer between some of the processors. But maybe this is a desired outcome.

I would also like to add that for now we are using the sample credentials to get FHIR token which is required for those two flows. I’m not sure, for what credentials should they be changed in the future.

Best,
Paulina.

ohinc · May 29, 2020, 1:29pm

Hi all,

Following the step-by-step instruction in addition to getting familiar with issues resolved at this forum thread should improve the process a lot, but there are still improvements to be made in order to simplify and shorten it. So, as discussed at the Technical Committee Call, the potential improvements for the reporting stack setup are described below.

First of all, there are some differences between the implementation of the reporting stack for the Core than for the implementations - there are different scripts, docker files, and other configuration files. Additionally the sources of restricted files like credentials, certificates or flows snapshots are stored in different places - on Jenkins, AWS s3 buckets, or on the repository. As a part of setting up the reporting stack for the COVID instance, we’ve unified all of those differences to have a working instance but without any implementation-specific changes. So our first improvement would be creating a reporting stack sample repository based on what we’ve just prepared for the COVID instance (or adjusting Core one and making sure it’s working well) as a startup point containing all initial files needed to set up a working reporting stack in one place, with some additional information of which values should be changed (like credentials, URLs, etc.).

Another improvement, strictly related to the previous one, would be to get rid of all implementation-related files in the distro repository, which would simplify its usage. Those files would be:

.env - configuration constants (it do not require any changes for a new implementation, but can be adjusted if needed)
NiFi-flows - those files are used as a source of the templates, but are adjusted during the setup (as while processing the values like identifiers (processor id, group id et.) are replaced by the NiFi)
Superset related files - these files are different in almost every implementation - there are assets to override the superset behavior and UI, and all data sources and dashboards configured
Any other files - as the implementations might have special requirements almost all files might needs adjustments for those (e.g. docker files or some particular scripts)

As there are a lot of files that are often changed, this repository is cloned for each implementation and adjusted separately there. Maybe a solution would be to move files that are always changed to a different place and allow to just override any other files only when there’s a need.

Another improvement that we’ve already considered as must-have is to replace all hardcoded values in the NiFi flows (@Paulina_Buzderewicz is already working on this).

Additional improvements could be:

Using terraform for setting up the AWS instance and for docker certificates could save up to a few hours.
Improve the docker files and scripts to make more actions automatically (like updating the superset requirements versions, creating the superset user in the database or initial URL/credentials propagation/generation). Also, we could investigate how to prevent superset from requiring the newest versions of the tools to avoid this issue.
Using the NiFi and NiFi registry REST API for managing the templates and the versions of process groups but this part is not consuming much time so we’re not sure if it’s worth investing time. The actions to be eventually replaced by the API calls would be:
1. Creating bucket on NiFi registry
2. Uploading templates config as XML files
3. Creating templates on the UI
4. Starting version controls for all
5. Exporting updated templates as XML files
6. Creating files with the flow is and versions for each process group

If you have any additional ideas, please share them as well.

Best,
Oskar

ohinc · June 2, 2020, 11:59am

Hi all,

Another challenge one might encounter during the setup is cache of SUPERSET_URL environmental variable. Please not that if its value is changed, the cache has to be cleared in order to allow using Superset from the OpenLMIS properly.

Best,
Oskar

Sebastian_Brudzinski · June 2, 2020, 1:59pm

One thing that I’ll add and that I keep hearing is that the users who want to access Superset reports need to have their e-mail address set in OpenLMIS (as this is required during auth to Superset). Setting e-mail address in OpenLMIS is not required for users, so this causes friction, especially for some demo users or other tests where you quickly want to spin up the instance, set up a user and test something.

Best,
Sebastian

ohinc · June 2, 2020, 3:19pm

Hi all,

Message ‘No data’ displayed on the Superset (like on the image below) might indicate that the reporting database has no records yet.

To load the data press stop and then start on each processor individually (after the previous one is finished)

Best,
Oskar

ohinc · June 4, 2020, 12:43pm

Hi all,

The estimations of the improvements for the Reporting Stack can be found int his document.

Best,
Oskar