at the 2020 March 11 Technical Committee meeting, we were discussing invalid token issues spotted both in core and Malawi implementation.
It shows up as endpoint calls response with an error message which indicates an invalid token as a cause.
The bug appears under huge workload i.e. performance tests or Malawi instance normal usage. At Malawi, the update of orderables increased the occurrence of this error. We are currently investigating it and for now, we know that:
- this is a service level authorization token,
- the token itself is valid.
We have increased the logging level for the Spring OAuth but it didn’t show any useful information.
Our next steps are:
- increasing the logging level for the Spring OAuth for Malawi,
- try to increase the pool size of the JDBC connection (link).
Most likely it is not related to token expiration. What do you think might be the root cause of the issue?
This is an old issue indeed. I think you’re on the right track. Aside from decreasing the pool size (I presume you meant decrease as the article mentions - I would be cautious with playing the increasing game with Postgres connections), have we checked if Stock Management service is acquiring a service token (access token) for every message (reference issue from Requisition service) still?
Hi @joshzamor, @wbulawa
After investigating the issue more deeply we couldn’t find a specific single cause of the issue - the token was correct on both client and server sides, but with a huge server load with simultaneous requests, it was still randomly considered invalid. Changing the pool size has no effect here. For stock management, the issue occurred more often because of a lack of caching it. However, we’ve applied a solution described below:
- If the invalid token error occurs, it’s verified again: for really invalid tokens the result will be the same, but correct tokens should be considered valid after retry
- The numbers of retry is configurable
- The solution was applied to each service
- Token caching was added to the stock management
After applying those changes we didn’t observe any more of the invalid_token errors, so I believe we can consider this topic closed.