Project UDFs Cache Settings
This page outlines the configuration for setting up project UDFs, which allow users to set their current project in Immuta through Spark. For details about the specific functions available and how to use them, see the Use Project UDFs (Databricks) page.
Use project UDFs in Databricks Spark
Currently, caches are not all invalidated outside of Databricks because Immuta caches information pertaining to a user's current project. Consequently, this feature should only be used in Databricks.
Lower the web service cache timeout in Immuta:
Click the App Settings icon and scroll to the HDFS Cache Settings section.
Lower the Cache TTL of HDFS user names (ms) to 0.
Click Save.
Raise the cache timeout on your Databricks cluster: In the Spark environment variables section, set the
IMMUTA_CURRENT_PROJECT_CACHE_TIMEOUT_SECONDS
andIMMUTA_PROJECT_CACHE_TIMEOUT_SECONDS
to high values (like10000
).Note: These caches will be invalidated on cluster when a user calls
immuta.set_current_project
, so they can effectively be cached permanently on cluster to avoid periodically reaching out to the web service.
Last updated
Was this helpful?