This document details how to externally generate native views for databases that are currently not supported as integrations in Immuta using the Immuta API, equalized projects, and webhooks.
Introduction
Immuta supports several integrations that allow users to interact directly with their systems without going through a proxy. For systems that Immuta does not currently support as an integration, users can generate native views for these systems using
The implementation of generating native views will vary based on internal business requirements; however, the general workflow and interaction with the Immuta API and webhooks are outlined below.
Webhooks
A number of webhooks ensure native views represent the current state of the policies enforced. The table below describes each of these notification types.
Notification Type
Description
dataSourceUpdated
This notification is triggered whenever a data source is updated, such as when it is enabled, disabled, or renamed; connection information or columns change; or metadata associated with the source changes.
addedToProject
This notification is triggered when a data source is added to a project.
removedFromProject
This notification is triggered when a data source is removed from a project.
policyUpdated
This notification is triggered whenever there is a policy change.
Register for webhook notifications
To receive webhook notifications, you must register for the notifications as a user with the APPLICATION_ADMIN permission to create a global webhook. Unlike a regular webhook, a global webhook ensures that you receive all notifications, even if they don’t pertain to the particular user who registered the hook. Note: You should only register the hook once.
All user-configured webhook integrations must respond within 10 seconds of receiving the webhook request payload. If the webhook integration takes longer to respond, the request will timeout.
Determine if the webhook exists
Before registering a webhook, check if the webhook already exists by comparing it to the names and URLs of existing webhooks. If there is already a webhook with the name and url of the one you want to register, it already exists and you don't need to register it. Completing this process whenever your service starts up will ensure that you never miss notifications.
Use the request and example response provided below for guidance:
Once a notification is received, additional calls need to be made to the Immuta API to ensure the native views reflect the current state of policies, data sources, and projects. These calls depend on the notification received, so the calls required for each are outlined by notification type throughout this section.
A data source has been updated
Use the tabs below for guidance when you receive the dataSourceUpdated notification.
When the dataSourceUpdated notification is triggered, the columns, credentials, or even the backing query of a data source may have changed or the data source may have been disabled or enabled. When a dataSourceUpdated notification is received,
Get the data source metadata to verify that the updated data source belongs to an equalized project. (See the 1. Get Data Source Metadata tab.)
Note: Notifications are not filtered by blobHandlerType, so you may receive notifications for irrelevant data source types. Whenever you receive this notification, check the returned blobHandlerType when you call the data source endpoint.
Get a list of equalized projects that contain this data source. (See the 2. Get a List of Relevant Projects tab.)
Regenerate the view for each of those projects by calling the view SQL endpoint:
Remove the view if the source is disabled.
Create the view (if the data source was disabled but has now been enabled).
Update the view (if some other update caused the view to change).
Call the project search endpoint and pass the data source id included in the data source metadata response.
Note: This API call should be made as a user with the APPLICATION_ADMIN permission so that all matching projects are returned regardless of subscription status.
Regenerate the view for each of those projects by calling the view SQL endpoint:
{"hits": [ {"id":9,"name":"Regulated Clients","status":"open","description":"A project containing data for regulated clients","deleted":false,"updatedAt":"2021-01-15T11:10:23.349Z","subscriptionPolicy":null,"createdAt":"2021-01-11T15:49:03.146Z","filterId":29,"subscriptionType":"manual","isEqualized":true,"acknowledgeRequired":false,"subscriptionStatus":"owner","purposeCount":0,"hasDeletedPurposes":false,"workspace":null,"dataSourceReason":null,"dataSourceAddedAt":"2021-01-13T17:04:51.941Z","datasourceAddedBy":"John Doe","allowMaskedJoins":true }, {"id":3,"name":"All Clients","status":"open","description":"A project containing data for current clients","deleted":false,"updatedAt":"2021-01-12T11:05:54.317Z","subscriptionPolicy":null,"createdAt":"2021-01-11T09:30:12.843Z","filterId":13,"subscriptionType":"manual","isEqualized":true,"acknowledgeRequired":true,"subscriptionStatus":"owner","purposeCount":1,"hasDeletedPurposes":false,"workspace":null,"dataSourceReason":null,"dataSourceAddedAt":"2021-01-11T09:30:13.215Z","datasourceAddedBy":"John Doe","allowMaskedJoins":true } ]}
A data source has been added to a project
Use the tabs below for guidance when you receive the addedToProject notification.
When a data source is added to a project,
Get the data source metadata to verify that the data source is queryable by checking its blobHandlerType. (See the 1. Get Data Source Metadata tab.)
Get the project metadata and confirm that the project is equalized: isEqualized: true. (See the 2. Get Project Metadata tab.)
Call the view SQL endpoint to generate the SQL for the native view.
When you receive the policyUpdated notification, determine if the policy has been updated for a project or data source by checking the modelType property in the notification, and then navigate to the corresponding section below for guidance.
modelType: dataSource
If the notification is for a data source,
Get the data source metadata and check the blobHandlerType to ensure that the data source is queryable. (See the 1. Get Data Source Metadata tab.)
Get a list of equalized projects that contain the data source. (See the 2. Get a List of Relevant Projects tab.)
Once you have the list of projects, make API calls to the view SQL endpoint for the impacted data source for each of the projects. For example, if the data source has an id of 11 and there were 4 projects with ids 1,4,5, and 9, you would need to make 4 API calls to generate the SQL for the native views.
Call the project search endpoint and pass the data source id from the notification. Note: This API call should be made as a user with the APPLICATION_ADMIN permission so that all matching projects are returned regardless of subscription status.
Make API calls to the view SQL endpoint for the impacted data source for each of the projects to regenerate the views.
{"hits": [ {"id":9,"name":"Regulated Clients","status":"open","description":"A project containing data for regulated clients","deleted":false,"updatedAt":"2021-01-15T11:10:23.349Z","subscriptionPolicy":null,"createdAt":"2021-01-11T15:49:03.146Z","filterId":29,"subscriptionType":"manual","isEqualized":true,"acknowledgeRequired":false,"subscriptionStatus":"owner","purposeCount":0,"hasDeletedPurposes":false,"workspace":null,"dataSourceReason":null,"dataSourceAddedAt":"2021-01-13T17:04:51.941Z","datasourceAddedBy":"John Doe","allowMaskedJoins":true }, {"id":3,"name":"All Clients","status":"open","description":"A project containing data for current clients","deleted":false,"updatedAt":"2021-01-12T11:05:54.317Z","subscriptionPolicy":null,"createdAt":"2021-01-11T09:30:12.843Z","filterId":13,"subscriptionType":"manual","isEqualized":true,"acknowledgeRequired":true,"subscriptionStatus":"owner","purposeCount":1,"hasDeletedPurposes":false,"workspace":null,"dataSourceReason":null,"dataSourceAddedAt":"2021-01-11T09:30:13.215Z","datasourceAddedBy":"John Doe","allowMaskedJoins":true } ]}
modelType: project
If the notification is for a project,
Get the project metadata and confirm that the project is equalized: isEqualized: true. (See the 1. Get Project Metadata tab.)
Get a list of all data sources contained in the project, and then filter these sources based on the blobHandlerType of each source. (See the 2. Get a List of Relevant Data Sources tab.)
Call the view SQL endpoint to regenerate the native views for these data sources.
Use the project data sources endpoint and specify the project id from the notification. Note: This API call should be made as a user with the APPLICATION ADMIN permission so that all matching projects are returned regardless of subscription status.
Filter the results based on the blobHandlerType value.
Call the view SQL endpoint to regenerate the native views for these data sources.