> For the complete documentation index, see [llms.txt](https://documentation.immuta.com/2024.3/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.immuta.com/2024.3/integrations/databricks-spark/reference-guides/configuration-settings/security-config-for-performance.md). # Databricks Security Configuration for Performance This page describes how the Security Manager is disabled for Databricks clusters that do not allow R or Scala code to be executed. Databricks Administrators should place the desired configuration in the Spark environment variables (recommended) or `immuta_conf.xml` (not recommended). ## Automatic Disabling of the Security Manager The Immuta Security Manager is an essential element of the Databricks Spark deployment that ensures users can't perform unauthorized actions when using Scala and R, since those languages have features that allow users to circumvent policies without the Security Manager enabled. However, the Security Manager must inspect the call stack every time a permission check is triggered, which adds overhead to queries. To improve Immuta's query performance on Databricks, Immuta disables the Security Manager when Scala and R are not being used. The cluster init script checks the cluster’s configuration and automatically removes the Security Manager configuration when * `spark.databricks.repl.allowedlanguages` is a subset of `{python, sql}` * `IMMUTA_SPARK_DATABRICKS_PY4J_STRICT_ENABLED` is `true` When the cluster is configured this way, Immuta can rely on Databricks' process isolation and Py4J security to prevent user code from performing unauthorized actions. *Note: Immuta still expects the `spark.driver.extraJavaOptions` and `spark.executor.extraJavaOptions` to be set and pointing at the Security Manager.* Beyond disabling the Security Manager, Immuta will skip several startup tasks that are required to secure the cluster when Scala and R are configured, and fewer permission checks will occur on the Driver and Executors in the Databricks cluster, reducing overhead and improving performance. ## Caveats * There are still cases that require the Security Manager; in those instances, Immuta creates a fallback Security Manager to check the code path, so the `IMMUTA_INIT_ALLOWED_CALLING_CLASSES_URI` environment variable must always point to a valid calling class file. * Databricks’ `dbutils.fs` is blocked by their `PY4J` security; therefore, it can’t be used to access scratch paths. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://documentation.immuta.com/2024.3/integrations/databricks-spark/reference-guides/configuration-settings/security-config-for-performance.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.