Setting Up Domains for Marketplace
Typically, you would give a data product manager CREATE permission in a schema or database that they can use as their sandbox for generating new tables/views natively in their data platform using data engineering tools like dbt. Those newly generated tables/views (or even S3 objects) are what they can use as the data sources for their data products.
You must get these new data objects from the data platform as registered in Immuta and assigned to a domain so that they can be published in data products:
Immuta automatically registers objects through periodic polling (24 hours by default) to detect changes in the data platform and represent those changes in Immuta, as data sources. These checks can also be manually triggered.
Once the objects are registered in Immuta as data sources, they are assigned to a domain one of two ways:
Manually: The data source is assigned to the domain through the Governance app (or API) by a user with
GOVERNANCEpermission.Dynamically (recommended): The data source is automatically assigned to the domain based on if it has a specific tag.
Tags can be applied directly to the tables/views in the data platform (Snowflake, Databricks Unity Catalog, and AWS Lake Formation only), imported from a supported external catalog, or applied through the Immuta UI.
See the examples in the tabs below to understand your options when dynamically assigning data sources to domains for data products.
Requirement: Data sources from a connection
An administrator of the data platform GRANTs CREATE permission to the hypothetical schema
business.hr-data-productsto the data engineers.User with
GOVERNANCEpermission creates the domainHR Domainand selects dynamic assignment based on the tagImmuta Connections . Snowflake . business . hr-data-products.User with
USER_ADMINpermission provides the data engineers with permissionManage Data Productsin that domain.Data engineer creates 6 new tables in the schema
business.hr-data-productsand wants to now have them available as data sources for a data product.When Immuta registers those objects, it will include the connection tag to represent the schema and database.
If Immuta hasn't yet found those new tables through periodic polling, the data engineer executes object sync over the Immuta API so that Immuta will find them.
Those 6 tables will appear as data sources within the domain and are now available for data products.
Requirement: Snowflake, Databricks Unity Catalog, or AWS Lake Formation data sources
An administrator of the data platform GRANTs CREATE permission to the hypothetical schema
business.hr-data-productsto the data engineers. This administrator also creates the tagHR Domainin the data platform to tag the tables.User with the
APPLICATION_ADMINpermission configures Snowflake, Databricks Unity Catalog, or AWS Lake Formation to ingest tags.User with
GOVERNANCEpermission creates the domainHR Domainand selects dynamic assignment based on the tagHR Domain.User with
USER_ADMINpermission provides the data engineers with permissionManage Data Productsin that domain.Data engineer creates 6 new tables in the schema
business.hr-data-productsand wants to now have them available as data sources for a data product.Data engineer tags those data sources with the
HR Domaintag directly in the data platform. When Immuta registers those objects, it will include the data platform tag(s).If Immuta hasn't yet found those new tables through periodic polling, the data engineer executes schema monitoring over the Immuta API so that Immuta will find them.
Those 6 tables will appear as data sources within the domain and are now available for data products.
An administrator of the data platform GRANTs CREATE permission to the hypothetical schema
business.hr-data-productsto the data engineers. This administrator also creates the tagHR Domainin the data platform to tag the tables.User with
GOVERNANCEpermission creates the new tagHR Domain.User with
GOVERNANCEpermission creates the domainHR Domainand selects dynamic assignment based on the tagHR Domain.User with
GOVERNANCEpermission configures the data engineers to be data owners of all the tables in the schemabusiness.hr-data-products(includes future tables). Being the data owner allows you to manage tags on the tables in the Governance app.Data engineer creates 6 new tables in the schema
business.hr-data-productsand wants to now have them available as data sources for a data product.If Immuta hasn't yet found those new tables through periodic polling, the data engineer executes schema monitoring over the Immuta API so that Immuta will find them.
Data engineer tags those data sources with the
HR Domaintag from within the Governance app (or with the API).Those 6 tables will appear as data sources within the domain and are now available for data products.
As you can see in all the examples, the GOVERNANCE user was able to still limit what data sources land in the HR Domain by limiting the scope of power where the data engineer could apply tags. In the first two examples, they are limited to applying tags only in the schema where they have CREATE permission in the data platform. In the second example, they are limited to where they can apply tags by where they were made data owners.
Last updated
Was this helpful?