> For the complete documentation index, see [llms.txt](https://documentation.immuta.com/latest/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.immuta.com/latest/configuration/integrations/databricks/databricks-spark/reference-guides/databricks/ephemeral-overrides.md). # Ephemeral Overrides In the context of the Databricks Spark integration, Immuta uses the term *ephemeral* to describe data sources where the associated compute resources can vary over time. This means that the compute bound to these data sources is not fixed and can change. All Databricks data sources in Immuta are ephemeral. Ephemeral overrides are specific to each data source and user. They effectively bind cluster compute resources to a data source for a given user. Immuta uses these overrides to determine which cluster compute to use when connecting to Databricks for various maintenance operations. The operations that use the ephemeral overrides include * **Visibility checks on the data source for a particular user**. These checks assess how to apply row-level policies for specific users. * **Stats collection triggered by a specific user**. * **Validating a custom WHERE clause policy against a data source**. When owners or governors create custom WHERE clause policies, Immuta uses compute resources to validate the SQL in the policy. In this case, the ephemeral overrides for the user writing the policy are used to contact a cluster for SQL validation. * **High cardinality column detection**. Certain advanced policy types (e.g., minimization) in Immuta require a high cardinality column, and that column is computed on data source creation. It can be recomputed on demand and, if so, will use the ephemeral overrides for the user requesting computation. ### Triggering an ephemeral override request An ephemeral override request can be triggered when a user queries the securable corresponding to a data source in a Databricks cluster with the Spark plug-in configured. The actual triggering of this request depends on the [configuration settings](/latest/configuration/integrations/databricks/databricks-spark/reference-guides/databricks/configuration.md#immuta_ephemeral_host_override). Ephemeral overrides can also be set for a data source in the Immuta UI by navigating to the data source page, clicking on the data source actions button, and selecting **Ephemeral overrides** from the dropdown menu. Ephemeral override requests made from a cluster for data sources and users where ephemeral overrides were set in the UI will not be successful. If ephemeral overrides are never set (either through the user interface or the cluster configuration), the system will continue to use the connection details directly associated with the data source, which are set during [data source registration](/latest/configuration/integrations/databricks/databricks-spark/reference-guides/registering-and-protecting-data.md#registering-data). ## Configuring overrides in Immuta-enabled clusters Ephemeral overrides can be problematic in environments that have a dedicated cluster to handle maintenance activities, since ephemeral overrides can cause these operations to execute on a different cluster than the dedicated one. To reduce the risk that a user has overrides set to a cluster (or multiple clusters) that aren't currently up, complete one of the following actions: * Direct all clusters' HTTP paths for overrides to a cluster dedicated for metadata queries using the [`IMMUTA_EPHEMERAL_HOST_OVERRIDE_HTTPPATH` Spark environment variable](/latest/configuration/integrations/databricks/databricks-spark/reference-guides/databricks/configuration.md#immuta_ephemeral_host_override_httppath). * Disable ephemeral overrides completely by setting the [`IMMTUA_EPHEMERAL_HOST_OVERRIDE` Spark environment variable](/latest/configuration/integrations/databricks/databricks-spark/reference-guides/databricks/configuration.md#immuta_ephemeral_host_override) to `false`. {% hint style="info" %} **Ephemeral overrides best practices** 1. Disable ephemeral overrides for clusters when using multiple workspaces and dedicate a single cluster to serve queries from Immuta in a single workspace. 2. If you use multiple E2 workspaces without disabling ephemeral overrides, avoid applying the **where user** row-level policy to data sources. {% endhint %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://documentation.immuta.com/latest/configuration/integrations/databricks/databricks-spark/reference-guides/databricks/ephemeral-overrides.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.