Further requisition endpoints performance improvements

Hello everyone,

  we are still investigating performance issues with requisition endpoints and are currently especially interested in requisition search endpoint. We have already looked into reducing the response size, but other than that, the requisition search endpoint makes a lot of calls to the reference data service, mainly to check user rights. The current workflow is: for each found requisition (before pagination) verify user rights, by sending 3 calls to the referencedata service - retrieve user, retrieve right, call 'hasRight' endpoint.

  In order to reduce the number of calls made to the reference data service, we are proposing the following:

  1. Be smarter about right checking. If we have multiple requisitions coming from the search, let's send an actual request to reference data only for program-facility combo that we didn't ask for already. This could be achieved by having a simple list that holds already checked combinations and checking with the list first, before we ask reference data service.

  2. Improve 'hasRight' endpoint contract. For some reason hasRight requires providing user UUID and right UUID in order to verify the user has necessary rights. Other services do not have access to user or right UUIDs, so they need to retrieve them from the reference data service first. Instead of doing all of that, the hasRight could just accept username and rightname what would make the usage of that endpoint simpler for clients.

  Do you have any comments or concerns about making those changes? The Malawi team would be up for implementing at least the first one. The second one may take some more time as there are many clients of that endpoint already, but if time allows we could get to that as well.

Best regards,

  Sebastian.
···


Sebastian Brudziński

    Software Developer / Team Leader

sbrudzinski@soldevelo.com

SolDevelo
Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

I don’t think number two will require necessarily changing the endpoint, we could just add a second one that is basically more of a search based endpoint - it can take multiple params like username, user id, right name, right id (and possibly deprecating the second one if need be)


SolDevelo
Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

···

On Wed, Jun 14, 2017 at 3:58 PM, Sebastian Brudziński sbrudzinski@soldevelo.com wrote:

Hello everyone,

  we are still investigating performance issues with requisition endpoints and are currently especially interested in requisition search endpoint. We have already looked into reducing the response size, but other than that, the requisition search endpoint makes a lot of calls to the reference data service, mainly to check user rights. The current workflow is: for each found requisition (before pagination) verify user rights, by sending 3 calls to the referencedata service - retrieve user, retrieve right, call 'hasRight' endpoint.
  In order to reduce the number of calls made to the reference data service, we are proposing the following:
  1. Be smarter about right checking. If we have multiple requisitions coming from the search, let's send an actual request to reference data only for program-facility combo that we didn't ask for already. This could be achieved by having a simple list that holds already checked combinations and checking with the list first, before we ask reference data service.
  2. Improve 'hasRight' endpoint contract. For some reason hasRight requires providing user UUID and right UUID in order to verify the user has necessary rights. Other services do not have access to user or right UUIDs, so they need to retrieve them from the reference data service first. Instead of doing all of that, the hasRight could just accept username and rightname what would make the usage of that endpoint simpler for clients.
  Do you have any comments or concerns about making those changes? The Malawi team would be up for implementing at least the first one. The second one may take some more time as there are many clients of that endpoint already, but if time allows we could get to that as well.

Best regards,

  Sebastian.


Sebastian Brudziński

    Software Developer / Team Leader


     sbrudzinski@soldevelo.com


SolDevelo
Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/751ceb7a-3e46-8221-0494-eec381f41470%40soldevelo.com.

For more options, visit https://groups.google.com/d/optout.

Paweł Gesek

    Technical Project Manager

     pgesek@soldevelo.com / +48 690 020 875

Apart from the ideas presented below, are we also queuing up a discussion about in-memory caching of reference data, either on startup for some of the smaller data sets, or lazily?
Reference Data won’t change much, so reducing the number of intra-service calls should also speed things up …

I realize that implementation of such caching won’t come in time for the MW release, but should be something on our list of enhancements.

···

From: <openlmis-dev@googlegroups.com<mailto:openlmis-dev@googlegroups.com>> on behalf of Paweł Gesek <pgesek@soldevelo.com<mailto:pgesek@soldevelo.com>>
Date: Wednesday, June 14, 2017 at 7:13 AM
To: Sebastian Brudziński <sbrudzinski@soldevelo.com<mailto:sbrudzinski@soldevelo.com>>
Cc: "openlmis-dev@googlegroups.com<mailto:openlmis-dev@googlegroups.com>" <openlmis-dev@googlegroups.com<mailto:openlmis-dev@googlegroups.com>>
Subject: Re: [openlmis-dev] Further requisition endpoints performance improvements

I don't think number two will require necessarily changing the endpoint, we could just add a second one that is basically more of a search based endpoint - it can take multiple params like username, user id, right name, right id (and possibly deprecating the second one if need be)

On Wed, Jun 14, 2017 at 3:58 PM, Sebastian Brudziński <sbrudzinski@soldevelo.com<mailto:sbrudzinski@soldevelo.com>> wrote:

Hello everyone,

we are still investigating performance issues with requisition endpoints and are currently especially interested in requisition search endpoint. We have already looked into reducing the response size, but other than that, the requisition search endpoint makes a lot of calls to the reference data service, mainly to check user rights. The current workflow is: for each found requisition (before pagination) verify user rights, by sending 3 calls to the referencedata service - retrieve user, retrieve right, call 'hasRight' endpoint.

In order to reduce the number of calls made to the reference data service, we are proposing the following:

1. Be smarter about right checking. If we have multiple requisitions coming from the search, let's send an actual request to reference data only for program-facility combo that we didn't ask for already. This could be achieved by having a simple list that holds already checked combinations and checking with the list first, before we ask reference data service.

2. Improve 'hasRight' endpoint contract. For some reason hasRight requires providing user UUID and right UUID in order to verify the user has necessary rights. Other services do not have access to user or right UUIDs, so they need to retrieve them from the reference data service first. Instead of doing all of that, the hasRight could just accept username and rightname what would make the usage of that endpoint simpler for clients.

Do you have any comments or concerns about making those changes? The Malawi team would be up for implementing at least the first one. The second one may take some more time as there are many clients of that endpoint already, but if time allows we could get to that as well.

Best regards,
Sebastian.

--

Sebastian Brudziński
Software Developer / Team Leader
sbrudzinski@soldevelo.com<mailto:sbrudzinski@soldevelo.com>

[http://www.soldevelo.com/sites/default/files/Soldevelo_logo_EPS_CMYK.png]
SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com<http://www.soldevelo.com>
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

--
You received this message because you are subscribed to the Google Groups "OpenLMIS Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com<mailto:openlmis-dev+unsubscribe@googlegroups.com>.
To post to this group, send email to openlmis-dev@googlegroups.com<mailto:openlmis-dev@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/751ceb7a-3e46-8221-0494-eec381f41470%40soldevelo.com<https://groups.google.com/d/msgid/openlmis-dev/751ceb7a-3e46-8221-0494-eec381f41470%40soldevelo.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

--

Paweł Gesek
Technical Project Manager
pgesek@soldevelo.com<mailto:pgesek@soldevelo.com> / +48 690 020 875

[http://www.soldevelo.com/sites/default/files/Soldevelo_logo_EPS_CMYK.png]
SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com<http://www.soldevelo.com>
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

--
You received this message because you are subscribed to the Google Groups "OpenLMIS Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com<mailto:openlmis-dev+unsubscribe@googlegroups.com>.
To post to this group, send email to openlmis-dev@googlegroups.com<mailto:openlmis-dev@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/CADt-Nu2axUrE%3DqwQM93F0d6ewnehQvTwDktOOS5iW-D3oGuHhg%40mail.gmail.com<https://groups.google.com/d/msgid/openlmis-dev/CADt-Nu2axUrE%3DqwQM93F0d6ewnehQvTwDktOOS5iW-D3oGuHhg%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

Regarding caching: Yes, caching reference data is part of the discussions and analysis happening now. In the very short term (ie, last 2 sprints), we are finding that we can make easy changes that take less programming time and have larger performance boosts. The most recent example is that we had 1000+ inter-service API calls happening that were reduced to one single call simply because we were making the calls individually in a loop rather than requesting a batch of records in a single API call. (Good work team on finding that using profiling tools and proposing an easy fix!)

Caching is discussed at many of the meetings/conversations we have about performance, so the time is coming when caching moves from discussion to action.

-Brandon

···

From: openlmis-dev@googlegroups.com on behalf of Jake Watson jake.watson@villagereach.org

Date: Wednesday, June 14, 2017 at 1:54 PM

To: Paweł Gesek pgesek@soldevelo.com, Sebastian Brudziński sbrudzinski@soldevelo.com

Cc:openlmis-dev@googlegroups.comopenlmis-dev@googlegroups.com

Subject: Re: [openlmis-dev] Further requisition endpoints performance improvements

Apart from the ideas presented below, are we also queuing up a discussion about in-memory caching of reference data, either on startup for some of the smaller data sets, or lazily?

Reference Data won’t change much, so reducing the number of intra-service calls should also speed things up …

I realize that implementation of such caching won’t come in time for the MW release, but should be something on our list of enhancements.

From: openlmis-dev@googlegroups.com on behalf of Paweł Gesek pgesek@soldevelo.com

Date: Wednesday, June 14, 2017 at 7:13 AM

To: Sebastian Brudziński sbrudzinski@soldevelo.com

Cc:openlmis-dev@googlegroups.comopenlmis-dev@googlegroups.com

Subject: Re: [openlmis-dev] Further requisition endpoints performance improvements

I don’t think number two will require necessarily changing the endpoint, we could just add a second one that is basically more of a search based endpoint - it can take multiple params like username, user id, right name, right id (and possibly deprecating the second one if need be)

On Wed, Jun 14, 2017 at 3:58 PM, Sebastian Brudziński sbrudzinski@soldevelo.com wrote:

Hello everyone,

we are still investigating performance issues with requisition endpoints and are currently especially interested in requisition search endpoint. We have already looked into reducing the response size, but other than that, the requisition search endpoint makes a lot of calls to the reference data service, mainly to check user rights. The current workflow is: for each found requisition (before pagination) verify user rights, by sending 3 calls to the referencedata service - retrieve user, retrieve right, call ‘hasRight’ endpoint.

In order to reduce the number of calls made to the reference data service, we are proposing the following:

  1. Be smarter about right checking. If we have multiple requisitions coming from the search, let’s send an actual request to reference data only for program-facility combo that we didn’t ask for already. This could be achieved by having a simple list that holds already checked combinations and checking with the list first, before we ask reference data service.
  1. Improve ‘hasRight’ endpoint contract. For some reason hasRight requires providing user UUID and right UUID in order to verify the user has necessary rights. Other services do not have access to user or right UUIDs, so they need to retrieve them from the reference data service first. Instead of doing all of that, the hasRight could just accept username and rightname what would make the usage of that endpoint simpler for clients.

Do you have any comments or concerns about making those changes? The Malawi team would be up for implementing at least the first one. The second one may take some more time as there are many clients of that endpoint already, but if time allows we could get to that as well.

Best regards,

Sebastian.

Sebastian Brudziński

Software Developer / Team Leader

sbrudzinski@soldevelo.com

**

SolDevelo** Sp. z o.o. [LLC] / www.soldevelo.com

Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland

Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to
openlmis-dev@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/openlmis-dev/751ceb7a-3e46-8221-0494-eec381f41470%40soldevelo.com
.

For more options, visit
https://groups.google.com/d/optout
.

Paweł Gesek

Technical Project Manager

pgesek@soldevelo.com / +48 690 020 875

**

SolDevelo** Sp. z o.o. [LLC] /
www.soldevelo.com

Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland

Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to
openlmis-dev@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/openlmis-dev/CADt-Nu2axUrE%3DqwQM93F0d6ewnehQvTwDktOOS5iW-D3oGuHhg%40mail.gmail.com
.

For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to
openlmis-dev@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/openlmis-dev/D566EF9E.F08F%25jake.watson%40villagereach.org
.

For more options, visit https://groups.google.com/d/optout.

I shared this idea this morning in a technical call on performance tickets, and I think while the suggestions above are valid (I have minor reservations about #2), we should also consider the following.

One way to flip this on its head is to attempt a concept that came out of a recent collaboration with ODK. In that collaboration it became apparent that we needed a way to translate our simple RBAC, which is mapped onto a complex Supervisory Node hierarchy, into a set of simple Strings which altogether described the User’s rights (aka permissions) in that hierarchy. Such a set of Permission Strings could, I believe, detail all of the different combinations of Rights that a User has on a Program and Facility.

Applied to the question posed here, database paging, there appears to be another use for this set of a User’s Permission Strings:

  • In order to achieve database level paging, filtering should be done in the database. More specifically, this means that we need to be able to express our “may User X do Y to Z” criteria directly in the databases’ criteria. This properly places the filtering work where it belongs and where it’s most effecient. RDBMS’s are made to index and filter, and we avoid the high expense of turning database rows into Java objects which we then filter and page on.
  • A row in the database, for example a Requisition, would need to express a set of Permission Strings that could be used.

The format of these permission strings should be consistent, and should be useful outside of Requisitions. A format which could work looks like the following:
program-facility-right

``

Where program, facility and right would be UUIDs.

Example

Lets create an example using this supervision structure:

Let’s assume the Program we’re interested in working with is for ARV, and the Facility W. Clinic is the one we want to see Requisitions at.

Requisition’s which were initiated at this facility would have a permission string such as:
ARV-W.CLINIC-viewRequisition

``

The Storeroom Assistant user has permission strings:

  • ARV-W.CLINIC-viewRequisition
  • ARV-W.CLINIC-createRequisition
  • MALARIA-W.CLINIC-viewRequisition
  • etc

The D. Supervisor user has permission strings:

  • ARV-W.CLINIC-viewRequisition
  • ARV-W.CLINIC-approveRequisition
  • ARV-E.CLINIC-viewRequisition
  • ARV-E.CLINIC-approveRequisition
  • MALARIA-D.HOSPITAL-createRequisition
  • etc

In our database paging of Requisition’s search, we now pass this set of User permission strings, and use something like a SQL IN to match. A simplified query would look something like:

SELECT * FROM requisitions WHERE permissionString IN (“ARV-W.CLINIC-viewRequisition”, “ARV-W.CLINIC-approveRequisition”, etc…)

``

We could even change the Permission Strings which the Requisition list as valid, as needed for filtering. e.g. when the requisition is authorized, we’d remove the string ARV-W.CLINIC-authorizeRequisition from the Requisition, and replace it with ARV-W.CLINIC-approveRequisition. This specific SQL may not work as well within Requisition Service, as the Requisition might want to express multiple permission strings: e.g. view and approve. An approach to this should be quite easy to tackle, even if it meant using another table to store the list of permissions for the requisition.

Other Role Assignments

Moving away from this example and looking at some of our design docs around RBAC, this format would work for the Supervisory Role Assignments (program-facility-right) used above as well as other types of assignment’s by dropping pieces off of the front of the string:

  • For a Fulfillment Role Assignment (aka Facility Role Assignment) we’d drop the program from the string to create: facility-right
  • For an Administrative/Report Role Assignment (aka Direct Role Assignment) we’d further drop the facility in addition to the program, and simply create: right

Of course there might be further optimization here that could be made.

  • These permission strings are actually UUIDs concatenated by hyphens - not truly strings we’d apply Regex to.
  • It also might make more sense to reverse the format, so that it’s: right-facility-program, and we’d drop pieces off the back of the string rather than the front for different types of Role Assignments.
  • Additionally we’d like to cache a User’s permission Strings for a period of time in consuming Services - any change in supervision structure, role definition, facility go-down date, etc would change this list, and we’d need to invalidate it in any caching strategy.
    Before we jump into all of that, I’d like to discuss the overall concept applied to database paging as well as to take a temperature check of the usefulness of this in other Services.

In summary:

  • The rights a user has, mapped by any current type of Role Assignment, could be expressed by the Reference Data Service as a set of Permission Strings.
  • An entity, such as a Requisition, could be searched for using the databases filtering capability more efficiently if we could compare the Permission Strings required for the entity to the Permission Strings a User has.
  • Permission Strings could be further optimized, however we’d like to first validate the concept.

I got in a SQL Fiddle mood: http://sqlfiddle.com/#!15/7d178/9

···

On Jun 14, 2017, at 3:58 PM, Josh Zamor josh.zamor@villagereach.org wrote:

I shared this idea this morning in a technical call on performance tickets, and I think while the suggestions above are valid (I have minor reservations about #2), we should also consider the following.

One way to flip this on its head is to attempt a concept that came out of a recent collaboration with ODK. In that collaboration it became apparent that we needed a way to translate our simple RBAC, which is mapped onto a complex Supervisory Node hierarchy, into a set of simple Strings which altogether described the User’s rights (aka permissions) in that hierarchy. Such a set of Permission Strings could, I believe, detail all of the different combinations of Rights that a User has on a Program and Facility.

Applied to the question posed here, database paging, there appears to be another use for this set of a User’s Permission Strings:

  • In order to achieve database level paging, filtering should be done in the database. More specifically, this means that we need to be able to express our “may User X do Y to Z” criteria directly in the databases’ criteria. This properly places the filtering work where it belongs and where it’s most effecient. RDBMS’s are made to index and filter, and we avoid the high expense of turning database rows into Java objects which we then filter and page on.
  • A row in the database, for example a Requisition, would need to express a set of Permission Strings that could be used.

The format of these permission strings should be consistent, and should be useful outside of Requisitions. A format which could work looks like the following:

program-facility-right

``

Where program, facility and right would be UUIDs.

Example

Lets create an example using
this supervision structure
:

Let’s assume the Program we’re interested in working with is for ARV, and the Facility W. Clinic is the one we want to see Requisitions at.

Requisition’s which were initiated at this facility would have a permission string such as:

ARV-W.CLINIC-viewRequisition

``

The Storeroom Assistant user has permission strings:

  • ARV-W.CLINIC-viewRequisition
  • ARV-W.CLINIC-createRequisition
  • MALARIA-W.CLINIC-viewRequisition
  • etc

The D. Supervisor user has permission strings:

  • ARV-W.CLINIC-viewRequisition
  • ARV-W.CLINIC-approveRequisition
  • ARV-E.CLINIC-viewRequisition
  • ARV-E.CLINIC-approveRequisition
  • MALARIA-D.HOSPITAL-createRequisition
  • etc

In our database paging of Requisition’s search, we now pass this set of User permission strings, and use something like a SQL IN to match. A simplified query would look something like:

SELECT

  • FROM requisitions WHERE permissionString IN (“ARV-W.CLINIC-viewRequisition”,
    “ARV-W.CLINIC-approveRequisition”, etc…)

``

We could even change the Permission Strings which the Requisition list as valid, as needed for filtering. e.g. when the requisition is authorized, we’d remove the string ARV-W.CLINIC-authorizeRequisition from the Requisition, and replace it with ARV-W.CLINIC-approveRequisition. This specific SQL may not work as well within Requisition Service, as the Requisition might want to express multiple permission strings: e.g. view and approve. An approach to this should be quite easy to tackle, even if it meant using another table to store the list of permissions for the requisition.

Other Role Assignments

Moving away from this example and looking at some of our design docs around RBAC, this format would work for the Supervisory Role Assignments (program-facility-right) used above as well as other types of assignment’s by dropping pieces off of the front of the string:

  • For a Fulfillment Role Assignment (aka Facility Role Assignment) we’d drop the program from the string to create: facility-right
  • For an Administrative/Report Role Assignment (aka Direct Role Assignment) we’d further drop the facility in addition to the program, and simply create: right

Of course there might be further optimization here that could be made.

  • These permission strings are actually UUIDs concatenated by hyphens - not truly strings we’d apply Regex to.
  • It also might make more sense to reverse the format, so that it’s: right-facility-program, and we’d drop pieces off the back of the string rather than the front for different types of Role Assignments.
  • Additionally we’d like to cache a User’s permission Strings for a period of time in consuming Services - any change in supervision structure, role definition, facility go-down date, etc would change this list, and we’d need to invalidate it in any caching strategy.
    Before we jump into all of that, I’d like to discuss the overall concept applied to database paging as well as to take a temperature check of the usefulness of this in other Services.

In summary:

  • The rights a user has, mapped by any current type of Role Assignment, could be expressed by the Reference Data Service as a set of Permission Strings.
  • An entity, such as a Requisition, could be searched for using the databases filtering capability more efficiently if we could compare the Permission Strings required for the entity to the Permission Strings a User has.
  • Permission Strings could be further optimized, however we’d like to first validate the concept.

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to
openlmis-dev@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/openlmis-dev/37eacf5b-c447-4785-9280-93c9c97e8100%40googlegroups.com
.

For more options, visit https://groups.google.com/d/optout.

This seems like an interesting idea; I wanted to clarify some things to make sure I understood it properly.

  • The permission strings that are associated with a requisition represent the rights that can be performed on that requisition, correct?
  • We would store these permission strings for a requisition in the Requisition Service, but based on what you are saying, they wouldn’t be stored in Reference Data, but generated each time the permissionStrings endpoint is called?
  • I assume program, facility and right would be UUIDs, which makes sense since I believe it is possible to change code strings for facilities and programs. You mention them as UUIDs, but the example has them as codes.

Shalom,

Chongsun

– ​

There are 10 kinds of people in this world; those who understand binary, and those who don’t.

Chongsun Ahn | chongsun.ahn@villagereach.org

Software Development Engineer

Village****Reach* ** Starting at the Last Mile*

2900 Eastlake Ave. E, Suite 230, Seattle, WA 98102, USA

DIRECT: 1.206.512.1536 **CELL: **1.206.910.0973 FAX: 1.206.860.6972

SKYPE: chongsun.ahn.vr

www.villagereach.org

Connect on Facebook****, Twitter** ** and our Blog

···

On Jun 14, 2017, at 3:58 PM, Josh Zamor josh.zamor@villagereach.org wrote:

I shared this idea this morning in a technical call on performance tickets, and I think while the suggestions above are valid (I have minor reservations about #2), we should also consider the following.

One way to flip this on its head is to attempt a concept that came out of a recent collaboration with ODK. In that collaboration it became apparent that we needed a way to translate our simple RBAC, which is mapped onto a complex Supervisory Node hierarchy, into a set of simple Strings which altogether described the User’s rights (aka permissions) in that hierarchy. Such a set of Permission Strings could, I believe, detail all of the different combinations of Rights that a User has on a Program and Facility.

Applied to the question posed here, database paging, there appears to be another use for this set of a User’s Permission Strings:

  • In order to achieve database level paging, filtering should be done in the database. More specifically, this means that we need to be able to express our “may User X do Y to Z” criteria directly in the databases’ criteria. This properly places the filtering work where it belongs and where it’s most effecient. RDBMS’s are made to index and filter, and we avoid the high expense of turning database rows into Java objects which we then filter and page on.
  • A row in the database, for example a Requisition, would need to express a set of Permission Strings that could be used.

The format of these permission strings should be consistent, and should be useful outside of Requisitions. A format which could work looks like the following:

program-facility-right

``

Where program, facility and right would be UUIDs.

Example

Lets create an example using
this supervision structure
:

Let’s assume the Program we’re interested in working with is for ARV, and the Facility W. Clinic is the one we want to see Requisitions at.

Requisition’s which were initiated at this facility would have a permission string such as:

ARV-W.CLINIC-viewRequisition

``

The Storeroom Assistant user has permission strings:

  • ARV-W.CLINIC-viewRequisition
  • ARV-W.CLINIC-createRequisition
  • MALARIA-W.CLINIC-viewRequisition
  • etc

The D. Supervisor user has permission strings:

  • ARV-W.CLINIC-viewRequisition
  • ARV-W.CLINIC-approveRequisition
  • ARV-E.CLINIC-viewRequisition
  • ARV-E.CLINIC-approveRequisition
  • MALARIA-D.HOSPITAL-createRequisition
  • etc

In our database paging of Requisition’s search, we now pass this set of User permission strings, and use something like a SQL IN to match. A simplified query would look something like:

SELECT

  • FROM requisitions WHERE permissionString IN (“ARV-W.CLINIC-viewRequisition”,
    “ARV-W.CLINIC-approveRequisition”, etc…)

``

We could even change the Permission Strings which the Requisition list as valid, as needed for filtering. e.g. when the requisition is authorized, we’d remove the string ARV-W.CLINIC-authorizeRequisition from the Requisition, and replace it with ARV-W.CLINIC-approveRequisition. This specific SQL may not work as well within Requisition Service, as the Requisition might want to express multiple permission strings: e.g. view and approve. An approach to this should be quite easy to tackle, even if it meant using another table to store the list of permissions for the requisition.

Other Role Assignments

Moving away from this example and looking at some of our design docs around RBAC, this format would work for the Supervisory Role Assignments (program-facility-right) used above as well as other types of assignment’s by dropping pieces off of the front of the string:

  • For a Fulfillment Role Assignment (aka Facility Role Assignment) we’d drop the program from the string to create: facility-right
  • For an Administrative/Report Role Assignment (aka Direct Role Assignment) we’d further drop the facility in addition to the program, and simply create: right

Of course there might be further optimization here that could be made.

  • These permission strings are actually UUIDs concatenated by hyphens - not truly strings we’d apply Regex to.
  • It also might make more sense to reverse the format, so that it’s: right-facility-program, and we’d drop pieces off the back of the string rather than the front for different types of Role Assignments.
  • Additionally we’d like to cache a User’s permission Strings for a period of time in consuming Services - any change in supervision structure, role definition, facility go-down date, etc would change this list, and we’d need to invalidate it in any caching strategy.
    Before we jump into all of that, I’d like to discuss the overall concept applied to database paging as well as to take a temperature check of the usefulness of this in other Services.

In summary:

  • The rights a user has, mapped by any current type of Role Assignment, could be expressed by the Reference Data Service as a set of Permission Strings.
  • An entity, such as a Requisition, could be searched for using the databases filtering capability more efficiently if we could compare the Permission Strings required for the entity to the Permission Strings a User has.
  • Permission Strings could be further optimized, however we’d like to first validate the concept.

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to
openlmis-dev@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/openlmis-dev/37eacf5b-c447-4785-9280-93c9c97e8100%40googlegroups.com
.

For more options, visit https://groups.google.com/d/optout.

Good questions, we’re also having some discussion and decisions in this ticket now: https://openlmis.atlassian.net/browse/OLMIS-2811

  • It’s up to the Requisition service to attach it’s meaning to any permission strings during authorization. The first use-case for this is to store the appropriate string(s) which powers a more efficient, database-paginated, /api/requisitions/search. It may be possible for the Requisition service to rely on permission strings for all / near all of it’s authorization checking, however that discovery is left for a later date.
  • The ticket OLMIS-2811 calls for Reference Data service to be the place that defines the format and generates the Set for any User. The performance criteria specified (including HTTP level caching support) do I think mean that the Reference Data service implementation will need a clever mechanism to store / cache the generated set, and properly respond to if-none-match in the HTTP header.
  • So far the consensus is to use UUIDs for everything, except the right name. The right name being more semantic and unchanging to a Service, this seems reasonable.
···

On Jul 18, 2017, at 9:42 PM, Chongsun Ahn chongsun.ahn@villagereach.org wrote:

This seems like an interesting idea; I wanted to clarify some things to make sure I understood it properly.

  • The permission strings that are associated with a requisition represent the rights that can be performed on that requisition, correct?
  • We would store these permission strings for a requisition in the Requisition Service, but based on what you are saying, they wouldn’t be stored in Reference Data, but generated each time the permissionStrings endpoint is called?
  • I assume program, facility and right would be UUIDs, which makes sense since I believe it is possible to change code strings for facilities and programs. You mention them as UUIDs, but the example has them as codes.

Shalom,

Chongsun

– ​

There are 10 kinds of people in this world; those who understand binary, and those who don’t.

Chongsun Ahn | chongsun.ahn@villagereach.org

Software Development Engineer

Village****Reach* ** Starting at the Last Mile*

2900 Eastlake Ave. E, Suite 230, Seattle, WA 98102, USA

DIRECT: 1.206.512.1536 **CELL: **1.206.910.0973 FAX: 1.206.860.6972

SKYPE: chongsun.ahn.vr

www.villagereach.org

Connect on Facebook****, Twitter** ** and our Blog

On Jun 14, 2017, at 3:58 PM, Josh Zamor josh.zamor@villagereach.org wrote:

I shared this idea this morning in a technical call on performance tickets, and I think while the suggestions above are valid (I have minor reservations about #2), we should also consider the following.

One way to flip this on its head is to attempt a concept that came out of a recent collaboration with ODK. In that collaboration it became apparent that we needed a way to translate our simple RBAC, which is mapped onto a complex Supervisory Node hierarchy, into a set of simple Strings which altogether described the User’s rights (aka permissions) in that hierarchy. Such a set of Permission Strings could, I believe, detail all of the different combinations of Rights that a User has on a Program and Facility.

Applied to the question posed here, database paging, there appears to be another use for this set of a User’s Permission Strings:

  • In order to achieve database level paging, filtering should be done in the database. More specifically, this means that we need to be able to express our “may User X do Y to Z” criteria directly in the databases’ criteria. This properly places the filtering work where it belongs and where it’s most effecient. RDBMS’s are made to index and filter, and we avoid the high expense of turning database rows into Java objects which we then filter and page on.
  • A row in the database, for example a Requisition, would need to express a set of Permission Strings that could be used.

The format of these permission strings should be consistent, and should be useful outside of Requisitions. A format which could work looks like the following:

program-facility-right

``

Where program, facility and right would be UUIDs.

Example

Lets create an example using
this supervision structure
:

Let’s assume the Program we’re interested in working with is for ARV, and the Facility W. Clinic is the one we want to see Requisitions at.

Requisition’s which were initiated at this facility would have a permission string such as:

ARV-W.CLINIC-viewRequisition

``

The Storeroom Assistant user has permission strings:

  • ARV-W.CLINIC-viewRequisition
  • ARV-W.CLINIC-createRequisition
  • MALARIA-W.CLINIC-viewRequisition
  • etc

The D. Supervisor user has permission strings:

  • ARV-W.CLINIC-viewRequisition
  • ARV-W.CLINIC-approveRequisition
  • ARV-E.CLINIC-viewRequisition
  • ARV-E.CLINIC-approveRequisition
  • MALARIA-D.HOSPITAL-createRequisition
  • etc

In our database paging of Requisition’s search, we now pass this set of User permission strings, and use something like a SQL IN to match. A simplified query would look something like:

SELECT

  • FROM requisitions WHERE permissionString IN (“ARV-W.CLINIC-viewRequisition”,
    “ARV-W.CLINIC-approveRequisition”, etc…)

``

We could even change the Permission Strings which the Requisition list as valid, as needed for filtering. e.g. when the requisition is authorized, we’d remove the string ARV-W.CLINIC-authorizeRequisition from the Requisition, and replace it with ARV-W.CLINIC-approveRequisition. This specific SQL may not work as well within Requisition Service, as the Requisition might want to express multiple permission strings: e.g. view and approve. An approach to this should be quite easy to tackle, even if it meant using another table to store the list of permissions for the requisition.

Other Role Assignments

Moving away from this example and looking at some of our design docs around RBAC, this format would work for the Supervisory Role Assignments (program-facility-right) used above as well as other types of assignment’s by dropping pieces off of the front of the string:

  • For a Fulfillment Role Assignment (aka Facility Role Assignment) we’d drop the program from the string to create: facility-right
  • For an Administrative/Report Role Assignment (aka Direct Role Assignment) we’d further drop the facility in addition to the program, and simply create: right

Of course there might be further optimization here that could be made.

  • These permission strings are actually UUIDs concatenated by hyphens - not truly strings we’d apply Regex to.
  • It also might make more sense to reverse the format, so that it’s: right-facility-program, and we’d drop pieces off the back of the string rather than the front for different types of Role Assignments.
  • Additionally we’d like to cache a User’s permission Strings for a period of time in consuming Services - any change in supervision structure, role definition, facility go-down date, etc would change this list, and we’d need to invalidate it in any caching strategy.
    Before we jump into all of that, I’d like to discuss the overall concept applied to database paging as well as to take a temperature check of the usefulness of this in other Services.

In summary:

  • The rights a user has, mapped by any current type of Role Assignment, could be expressed by the Reference Data Service as a set of Permission Strings.
  • An entity, such as a Requisition, could be searched for using the databases filtering capability more efficiently if we could compare the Permission Strings required for the entity to the Permission Strings a User has.
  • Permission Strings could be further optimized, however we’d like to first validate the concept.

You received this message because you are subscribed to the Google Groups “OpenLMIS Dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to
openlmis-dev@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/openlmis-dev/37eacf5b-c447-4785-9280-93c9c97e8100%40googlegroups.com
.

For more options, visit https://groups.google.com/d/optout.