Extension point configuration

At the moment we managed to verify the example extension point under versions 1.3, 1.5, and 2.2.2.
On all versions, we managed to run an example extension point on
api/extensionPoint
so we can exclude spring boot version as the reason for the problem.
In the case of openlmis-example version 2.2.2, to build the image we used gradle v6.5 with a wrapper by
./gradlew clean build
We tried a configuration in which we threw an extension point from openlmis-example into stockmanagement, but without a positive result.

The changes we have made to change the spring boot version in openlmis example are located on the branches:
OLMIS-6921-SB1.3.3
OLMIS-6921-1.5
OLMIS-6921 (for version 2.2.2)

Best,
Adrian

@joshzamor Yes, we probably should add this info somewhere but where is the best place? In the readme file?

Also, I’m adding more precise steps to reproduce the issue. To verify stock management extension point configuration you should use OLMIS-6911 branches of all mentioned GitHub repositories.

Start with stock management:

docker-compose run --service-ports stockmanagement
gradle clean build
exit

docker-compose -f docker-compose.builder.yml build image
docker tag openlmis/stockmanagement:latest openlmis/stockmanagement:5.1.2-SNAPSHOT

docker-compose run --service-ports stockmanagement
gradle fatJar
exit

cp build/libs/openlmis-stockmanagement-5.1.2-SNAPSHOT.jar openlmis-example-extension/libs/openlmis-stockmanagement-5.1.2-SNAPSHOT.jar

Example extension:

docker-compose -f docker-compose.yml run builder
cp build/libs/openlmis-example-extension-0.0.1-SNAPSHOT-sources.jar openlmis-example-extensions/libs/openlmis-example-extension-0.0.1-SNAPSHOT.jar

Example extensions:

docker-compose -f docker-compose.builder.yml run builder
docker-compose -f docker-compose.builder.yml build image
docker tag openlmis/openlmis-example-extensions:latest openlmis/openlmis-example-extensions:0.0.1-SNAPSHOT

Ref-distro:

docker-compose up

and check /api/extensionPoint via API definition: {your-ip}/stockmanagement/docs/.

Best,
Klaudia

1 Like

I spent some time digging into this yesterday, and I think the cause of the problem is that the example-extensions jar that is made available via docker volume and set as a runtime dependency, doesn’t make its way inside the service.jar

Taking a look at the behavior of the Spring Scanner, it scans two places:
service.jar!/BOOT-INF/classes - this contains our service source files
service.jar!/BOOT-INF/lib - this contains dependencies

Those are the log levels I used:

logging.level.org.springframework.beans.factory=DEBUG
logging.level.org.springframework.context.annotation=DEBUG
logging.level.org.springframework.core.io.support=DEBUG

The example-extensions jar is present inside the docker container via the docker volume named extensions but it doesn’t get included in the service.jar and thus isn’t discovered by the Spring Scanner when looking for spring beans.

It seems that for the example service, the example-extensions jar is properly placed inside the service.jar!/BOOT-INF/lib

If the above theory is true, the example service somewhere somehow has some magic that causes the example-extensions jar to be copied/moved inside the service.jar - the question remains where that magic is.

Best,
Sebastian

We managed to add an example-extension jar to service.jar file, the problem was the lack of volume extensions in the stockmanagement docker-compose file. However, we do not want extensions service in stockmanagement to be build-time dependency but run-time. Most likely the problem lies at the spring boot plugin level what we are trying to verify.

Best,
Adrian

Since the previous post we managed to verify the following things:

  • Removing uploadArchives task from the build.gradle in example service, to check if the example extension will still behave correctly - it still behaved correctly.

  • Adding
    java $JAVA_OPTS -classpath extensions/*:service.jar org.springframework.boot.loader.JarLauncher && java $JAVA_OPTS -jar service.jar
    to run.sh file in stockmanagement, following fix from this commit
    https://github.com/OpenLMIS/openlmis-example/commit/dec4e4bcf0916f2bb6f2bd9e6e1824fff40a5e93

  • Change service-base:4 in Dockerfile in stockmanagement to service-base:1 (as in example service)

We are still looking for a solution to this problem

We’ve also checked that removing run.sh from stock management doesn’t help. Moreover, even if the script in stock is different than it should, the proper version from openlmis-service-base v4 is used.

VERY IMPORTANT QUESTION

@joshzamor Do you happen to know if adding this dependency in example’s docker-compose file was intentional? If yes, should we create a public openlmis-stockmanagement-extensions and include it in stock management’s compose file? Otherwise, it means that the example also isn’t working properly with jars because we managed to verify that is stops working without the mentioned dependency.

Moreover, when we try to build the example (with the mentioned dependency) using the following commands, extensions don’t work (bean is not found):

docker-compose -f docker-compose.builder.yml run builder
docker-compose -f docker-compose.builder.yml build image

Usually, we use these:

docker-compose run --service-ports example
gradle clean build
exit
docker-compose -f docker-compose.builder.yml build image

Best,
Klaudia

Besides what Klaudia mentioned in the previous post we removed
maven { url "https://oss.sonatype.org/content/repositories/snapshots" }
in build.gradle in openlmis example to verify that the extension is not being downloaded from an external repository but a local jar. The extension point continued to work properly.

Best,
Adrian

Hi @Klaudia_Palkowska,

Do you happen to know if adding this dependency in example’s docker-compose file was intentional?

The docker-compose links is the old way that docker-compose v2 format linked containers. I’m seeing depends_on in your ref distro change and the containers are in the same network, so effectively you’ve done the same thing. example-extensions doesn’t really run, aside from moving the jar into the volume. So long as that’s done before stockmanagement service starts (which depends_on should achieve in terms of container creation – not starting), it should be the same difference.

@joshzamor My question was more about the correctness of such dependency in overall. Whether it should be added in example’s docker-compose.yml or not. We’ve verified example extension point configuration by running ref-distro with depends_on but without links in example service, and is not working.

We also found out that the example image contains an extension jar, and that’s why it looked like it was working.

Best,
Klaudia

So if I understand correctly, the example service extension only works because it is a build-time dependency of the example service, right? The available extensions are placed inside the example service during compilation. If I recall correctly, that is not desired, right @joshzamor?

Looking inside the running stock management service docker container, here’s how the structure looks like

service.jar/
    BOOT-INF/
        classes/
        lib/
extensions/
     example-extension-0.0.1.jar

The question is - is Spring Boot capable of finding spring beans in an external jar (not present in the service.jar) and if so, under what conditions?

If it isn’t, we can likely force the contents of the example-extension.jar inside the service.jar during container startup (i.e. as a part of run.sh).

Best,
Sebastian

That would certainly not be desired. Are we saying that we think this was true when we created the exemplar? Or that something has changed since then? Run-time, not build-time, was the objective. It’s been years but I’m certain about that.

It certainly is, which is why I’d be quite invested in understanding if something has changed since we built the example. The whole point of that exercise at that time was to show how to do it. Have we done any google searches for other examples? I just found this one but don’t have the time tonight to dive into it. Is that useful?

Thanks everyone for keeping at this. And sorry I haven’t been able to contribute more.

Currently, we have managed to check the example of dynamic dependency loading from GitHub unfortunately it did not work in our case. We also tried to add an executable jar extension point to service.jar in stockamangement but also without positive results. We are now continuing our attempts to add run time dependency of extension-point to stockmanagement.

Best,
Adrian

Based on the commit history, it doesn’t seem like anything has changed since then. It looks like the exemplar extension point might have never worked as expected. We likely need to drop looking at it and figure out a solution from scratch.

Best,
Sebastian

Could you post a github link to this jar @Sebastian_Brudzinski? When I run through @Klaudia_Palkowska’s steps, I don’t see this compile time dependency.

Could you post a github link to this jar @Sebastian_Brudzinski? When I run through @Klaudia_Palkowska’s steps, I don’t see a compile time dependency.

I made a couple of changes to example-extension (bump spring boot, gradle, build regular jar instead of bootJar, etc.) and we seem to be one step closer to fixing it.

The stock management service can finally find the extension jar, however it’s not able to load it due to missing base service classes on the classpath.

2020-10-13 12:55:28.413 DEBUG 35 — [ main] o.s.c.a.ClassPathBeanDefinitionScanner : Identified candidate component class: URL [jar:file:/extensions/openlmis-example-extension-0.0.1-SNAPSHOT.jar!/org/openlmis/stockmanagement/extensions/validators/TestValidator.class]

java.lang.IllegalStateException: Cannot load configuration class: org.openlmis.stockmanagement.extensions.validators.TestValidator
org.springframework.beans.factory.CannotLoadBeanClassException: Error loading class [org.openlmis.stockmanagement.extensions.validators.TestValidator] for bean with name ‘TestValidator’ defined in URL [jar:file:/extensions/openlmis-example-extension-0.0.1-SNAPSHOT.jar!/org/openlmis/stockmanagement/extensions/validators/TestValidator.class]: problem with class file or dependent class; nested exception is java.lang.NoClassDefFoundError: org/openlmis/stockmanagement/extension/point/AdjustmentReasonValidator

The problem is likely with this line in run.sh that is meant to provide the correct classpath - https://github.com/OpenLMIS/openlmis-service-base/blob/554dd0194c4afd91d6cc68b8aa97f2a00a887826/run.sh#L9

Hopefully, this is the last thing we need to fix to have that working.

Best,
Sebastian

Thanks @Sebastian_Brudzinski, for this and the discussion in the technical committee call today. After the call I remembered that Fulfillment service has an extension point, and that we might have another exemplar out there - a Malawi implementation that extended the order id generation apparently. A quick forum search pulled up this post on it. Is that helpful? And with this I’ll re-ask what I had on the call: how could we use this fulfillment extension and/or this new stock management one as our new exemplar and retire the example service?

Yes, Malawi has requested the extension point, but if I recall correctly they never got to implementing one. The change was either seen as globally applicable or it was made in a way that doesn’t require extensiomn point. Either way, Malawi is not using any backend extension modules at the moment.

I’d propose the following way forward to demonstrate the extension points mechanism

  1. Create new repo with an extension module that implements Stock’s ext point - AdjustmentReasonValidator. The repo would be called “openlmis-stockmanagement-validator-extension”
  2. The example-extensions points to and builds image with openlmis-stockmanagement-validator-extension instead of example-extension
  3. The Stock Management service publishes its code to Maven Central
  4. We add an alternative docker-compose file to openlmis-ref-distro that starts current ref-distro with extension module for Stock
  5. We drop example-extension repo (worth considering if openlmis-example is of any use to us as well)
  6. We add an automated test that starts ref-distro with extension module and verifies it works
  7. Review and update docs if needed

Thoughts?

1 Like

Hello,

The solution for our classpath issue was found. The bean from our extension jar is being registered now. In our run.sh script we used PropertiesLauncher instead of the JarLauncher. The line was changed from

java $JAVA_OPTS -classpath extensions/*:service.jar org.springframework.boot.loader.JarLauncher && java $JAVA_OPTS -jar service.jar

to

java -cp service.jar -Dloader.path=extensions/ org.springframework.boot.loader.PropertiesLauncher

It looks like, thanks to the PropertiesLauncher, such properties as -Dloader are taken into account making the bean from our additional jar successfully loaded.

Now, we can focus on taking further steps related to the extension point request.

Best,
Paulina.

1 Like