Setting Up Domains for Marketplace

Typically, you would give a data product manager CREATE permission in a schema or database that they can use as their sandbox for generating new tables/views natively in their data platform using data engineering tools like dbt. Those newly generated tables/views (or even S3 objects) are what they can use as the data sources for their data products.

You must get these new data objects from the data platform as registered in Immuta and assigned to a domain so that they can be published in data products:

• Immuta automatically registers objects through periodic polling (24 hours by default) to detect changes in the data platform and represent those changes in Immuta, as data sources. These checks can also be manually triggered.

• Once the objects are registered in Immuta as data sources, they are assigned to a domain one of two ways:

  • Manually: The data source is assigned to the domain through the Governance app (or API) by a user with GOVERNANCE permission.

  • Dynamically (recommended): The data source is automatically assigned to the domain based on if it has a specific tag.

    Tags can be applied directly to the tables/views in the data platform (Snowflake and Databricks Unity Catalog only), imported from a supported external catalog, or applied through the Immuta UI.

See the examples in the tabs below to understand your options when dynamically assigning data sources to domains for data products

Connections tags

Requirement: Data sources from a connection

1. An administrator of the data platform GRANTs CREATE permission to the hypothetical schema business.hr-data-products to the data engineers.

2. User with GOVERNANCE permission creates the domain HR Domain and selects dynamic assignment based on the tag Immuta Connections . Snowflake . business . hr-data-products.

3. User with USER_ADMIN permission provides the data engineers with permission Manage Data Products in that domain.

4. Data engineer creates 6 new tables in the schema business.hr-data-products and wants to now have them available as data sources for a data product.

5. When Immuta registers those objects, it will include the connection tag to represent the schema and database.

   1. If Immuta hasn't yet found those new tables through periodic polling, the data engineer executes object sync over the Immuta API so that Immuta will find them.

6. Those 6 tables will appear as data sources within the domain and are now available for data products.

Data platform tags

Requirement: Snowflake or Databricks Unity Catalog data sources

1. An administrator of the data platform GRANTs CREATE permission to the hypothetical schema business.hr-data-products to the data engineers. This administrator also creates the tag HR Domain in the data platform to tag the tables.

2. User with the APPLICATION_ADMIN permission configures Snowflake or Databricks Unity Catalog to ingest tags.

3. User with GOVERNANCE permission creates the domain HR Domain and selects dynamic assignment based on the tag HR Domain.

4. User with USER_ADMIN permission provides the data engineers with permission Manage Data Products in that domain.

5. Data engineer creates 6 new tables in the schema business.hr-data-products and wants to now have them available as data sources for a data product.

6. Data engineer tags those data sources with the HR Domain tag directly in the data platform. When Immuta registers those objects, it will include the data platform tag(s).

   1. If Immuta hasn't yet found those new tables through periodic polling, the data engineer executes schema monitoring over the Immuta API so that Immuta will find them.

7. Those 6 tables will appear as data sources within the domain and are now available for data products.

Immuta tags

1. An administrator of the data platform GRANTs CREATE permission to the hypothetical schema business.hr-data-products to the data engineers. This administrator also creates the tag HR Domain in the data platform to tag the tables.

2. User with GOVERNANCE permission creates the new tag HR Domain.
3. User with GOVERNANCE permission creates the domain HR Domain and selects dynamic assignment based on the tag HR Domain.
4. User with GOVERNANCE permission configures the data engineers to be data owners of all the tables in the schema business.hr-data-products (includes future tables). Being the data owner allows you to manage tags on the tables in the Governance app.
5. Data engineer creates 6 new tables in the schema business.hr-data-products and wants to now have them available as data sources for a data product.

   1. If Immuta hasn't yet found those new tables through periodic polling, the data engineer executes schema monitoring over the Immuta API so that Immuta will find them.
6. Data engineer tags those data sources with the HR Domain tag from within the Governance app (or with the API).

7. Those 6 tables will appear as data sources within the domain and are now available for data products.

As you can see in all the examples, the GOVERNANCE user was able to still limit what data sources land in the HR Domain by limiting the scope of power where the data engineer could apply tags. In the first two examples, they are limited to applying tags only in the schema where they have CREATE permission in the data platform. In the second example, they are limited to where they can apply tags by where they were made data owners.