Magda v2.0.0 the Separation of Authorization Enforcement and Decision-making in Microservices Architecture

7 min readAug 29, 2022

Magda is an open-source data catalog management solution used by many famous projects, including data.gov.au. v2.0.0 is the first major release since v1.0.0 was released last year. This release offers a complete decoupled authorization solution that separates the authorization Enforcement from the decision-making under the microservices architecture.

Why Decoupled Authorisation?

Authorization is hard. Not only because it is required to happen at so many places at all levels of your application. The complexity often also comes from the rapid changes in the authorization requirements. If we can’t separate the enforcement from the decision-making to create a decoupled authorization system, we can never guarantee a consistent & timely system-wide authorization policy roll-out.

The separation is the intrinsic need of microservice architecture as well. A loosely coupled microservice is often created in a “database per service” pattern and doesn’t have access to all data (e.g. user data) required for local authorization decision-making. Making some user attributes available to the services via JWT tokens is possible. But it would not be practical to store all user attributes in JWT tokens for complex user models considering the size of the JWT token and the overheads for every request.

Instead of making all required data available in the local data store (e.g. via “Materialized View Pattern” or similar techniques ) for authorization decisions, our solution is to leverage the partial evaluation feature of the policy engine “Open Policy Agent”.

In Magda, the authorization service will make a partial authorization decision by utilizing locally available user-related data when receiving a decision-sought request. Once receives the partial decision, the decision-sought service will convert it into storage engine DSL (domain-specific language) (e.g. SQL for database) for authorization enforcement.

Illustration of the Authorization Solution

In this way, we eliminate the need of aggregating data owned by different microservices for decision making and separate the decision making from enforcement.

Overview

Before v2.0.0, we had already introduced the Open Policy Agent (OPA) as the central policy engine serving authorization decisions at several API endpoints. However, our previous implementation didn’t come with an authorization model. Instead, we asked the user to supply his model via policy files and expected it would work with our enforcement & decision architecture.

This proved to be a mistake as our decision architecture can never be powerful enough to fully separate the enforcement from the decision-making without sufficient abstraction offered by the authorization model. Therefore, in v2.0.0, a brand new authorization model has been introduced. Users are still able to supply policy files to extend this model. To help users to manage the related authorization model objects, we also provided a “new setting panel” web UI.

Authorization Model

Our authorization model, firstly, defined the concept of “resources” & “operations”. A resource represents any access control targets. It can be a record, dataset, API, or UI component. A resource may support one or more “operations” that are the actions that could be performed on the resources.

Both “resources” & “operations” can be identified by URIs. e.g. the URI for record resource is object/record and it supports the following operations:

object/record/create
object/record/update
object/record/read
object/record/delete

The resource & operation URIs impact how we request authorization decision and organize our policy files.

Decision Architecture

In v2.0.0, we introduced a decision API as a new addition to the Auth API service. All decision requests should be sent to this API instead of querying the policy engine directly. e.g. When a user requests to read a “record”, we request a decision for operation URI object/record/read from the decision endpoint. Once the decision point receives the request, it will ask the policy engine to evaluate the “entry point policy” file. This “entry point” policy will then delegate the decision-making to the appropriate policy according to the requested operation URI.

Generally, we put a policy defined for a resource into the policy package corresponding to its resource URI. e.g. the policy defined for resource object/record should be put into policy package object.record. In this way, the the “entrypoint policy” can route the request to right policy simply by evaluate the policy in the corresponding package.

Generally, we always put a policy defined for a resource into the policy package corresponding to its resource URI. e.g. the policy defined for resource object/record should be put into the policy package object.record. In this way, the “entry point policy” can route the request to the right policy simply by evaluating the policy in the corresponding package.

The decision request routing mechanism enables a consistent decision-making interface regardless of the inclusive relationship between different types of resources. It avoids redundant authorization rules to ensure the single source of truth principle.

For instance, our metadata store service (the “registry”) allows users to create specialized records from generic records. e.g. an “order” record can be defined as a record attached with the “order-details” aspect. Our built-in dataset record is defined as a record with the attached “dcat-dataset-strings” aspect.

When the “entry point” policy is queried with operation URI object/dataset/read, the entry point will delegate the decision-making to policy in package object.dataset and output rules specific for dataset records.

However, when querying the “entry point” policy with operation URI object/record/read (i.e. the generic record resource concept), the entry point will delegate the decision to all defined subset resources policies (e.g. “dataset”, “distribution”, “organization” etc.). Together with authorization rules for generic record resources, the query result will be similar to the following pseudo code:

- when the record looks like a dataset record, the following rules apply.
- when the record looks like a distribution record, the following rules apply.
- when the record looks like a organization record, the following rules apply.
....
- otherwise, the common rules for generic record apply for any other type of records.

New Settings Panel

v2.0.0 also comes with a new settings panel for all signed-in users. The settings panel UI is supplied to simplify the everyday admin tasks such as users, roles, permissions, resources and operations management. Depending on your access, you might have access to different sections. e.g. all users should at least be able to access the “My Account” tab, and users with the admin role can access all tabs.

To assign the admin role to your first account, please refer to this doc to set a user as admin using the acs-cmd command line tool.

More Powerful APIs with fine-grained access control

Thanks to the new authorization system design. We are able to re-implement fine-grained access control to ALL our existing APIs.

Many APIs (e.g. indexer APIs) that previously were not accessible outside the cluster can be accessible outside the cluster as we can grant access to non-admin users that enable more use cases.

With the help of the new policy mode, we can still ensure all APIs are compatible with existing plugins & sub-systems. Although all requests between sub-systems are all governed by the policy engine now, requests from existing sub-systems will work as existing admin accounts still have access to any resources.

However, authors of existing plugins/sub-systems might consider updating the code and using an account with the least privilege to communicate with the core APIs, as it’s possible now with fine-grained access control of APIs.

Breaking Changes & Compatibility

Setting a policy for a record via the `authnReadPolicyId` field is not supported anymore.
Please manage access to registry records via built-in permission & constraints. See the permissions/role section of our architecture doc.
You can still supply your policy files to extend the authorization model. However, your policy must comply with the single entry point policy model. e.g. a policy governing resources `object/resourceB` should be put into package `object.resourceB`. You might also want to read the decision API doc
`isAdmin` field on the user will no longer grant the user admin permission.
All scripts/command line tools now require Node 14

Migration

If you are on version <= v1.3.1, you should be able to migrate to v2.0.0 simply by deploying the v2.0.0 Helm Chart. If you are on any v2 alpha versions (e.g. v2.0.0-alpha.8), you will have to completely uninstall Magda before upgrading to v2.0.0 as the SQL migration files won’t tell the difference between v2.0.0 and any of the v2 alpha versions.

Other Improvements

For a list of other improvements, please refer to the release notes.

Try it out

To try it out v2.0.0, you can access our Helm chart repo: https://charts.magda.io and access magda chart with version 2.0.0.

For developers, you can access npm for our latest SDK version 2.0.0.