All activity in Immuta is audited. This process provides rich audit logs that detail who subscribes to each data source , why they subscribe, when they access data, what SQL queries and blob fetches they run, and which files they access. Audit logs can be used for a number of intentions, including insider threat surveillance and data access monitoring for billing purposes. Audit logs can also be shipped to your enterprise auditing capability.
Best Practices: Store Audit Records
Store audit records outside of Immuta in order to retain the audit logs long-term.
By default, Immuta stores most audit records for 60 days. The following audit record types do not expire after 60 days:
blobFetch
dataSourceSubscription
globalPolicyApproved
globalPolicyApprovalRescinded
globalPolicyChangeRequested
globalPolicyConflictResolved
globalPolicyCreate
globalPolicyDelete
globalPolicyDisabled
globalPolicyUpdate
nativeQuery
policyExemption
policyHandlerCreate
policyHandlerUpdate
prestoQuery
spark
sqlQuery
Immuta writes all logs to stdout
in Kubernetes. Users will get all logs here, but should use the Common Message Types to parse for specifics.
When running Immuta on Docker-based installations, all logs from the Immuta Docker containers will be sent to the Docker log driver.
The Immuta log file will contain messages that are one-line JSON, as described in Log Formats.
Logs messages from the Immuta platform typically will be one line JSON and contain all of the common JSON properties. Depending on the message type, more JSON properties may be present. See Common Message Types for more details.
level: This is a string representation of the log level. Acceptable values are "debug", "info", "warning", "error", and "audit".
timestamp: This is a timestamp for when the message occurred. The timestamp format is YYYY-MM-DDTHH:mm:ss.sssZ
(ISO 8601).
message: This is the log message, which may be used to determine common message types.
Generally, any query that causes multiple background queries will have an audit record created for each. The audited plan should be different, however, for each record. For example, subqueries will generate two audit records: one for the subquery and one for the outer query.
Immuta does not audit any notebook cells that don’t query data.
Each audit message from the Immuta platform will be a one-line JSON object containing the common JSON properties and the Audit JSON properties. Depending on the recordType
, an audit message may contain additional data.
In order to discover audit messages using your analysis tool, you may search the object using the criteria below:
level: "audit"
message: "Audit - *
dateTime:
description: The timestamp for when the record was created. This may be an ISO-8601 timestamp string or a ms since epoch timestamp.
type: integer or string
example: 1504188066580
or "2017-08-31T14:01:15.607Z"
component:
description: The Immuta component that generated this record. Possible values are "console"
, "featureStore"
, "dataSource"
, "bim"
, "audit"
, "script"
, "policy"
, "project"
, "plugin"
, and "governance"
.
type: string
instanceId:
description: The instance ID of the component generating this record.
type: string
profileId:
description: The profile ID of the user generating the action.
type: integer
userId:
description: The user ID of the user generating the action.
type: string, null
sqlUser:
description: The database account generating the action.
type: string
dataAccess:
description: Describes access to an individual blob or a query that may grant access to multiple blobs.
type: object
sessionId:
description: If this record is generated in response to a user action and if that user's session ID is known, record that session ID here.
type : string
dataSource:
description: If the record creation is associated with a data source, the data source name should be recorded here.
type: string, null
dataSourceId:
description: If the record creation is associated with a data source, the data source ID should be recorded here.
type: integer, null
projectName:
description: If the record creation is associated with a project, the project name should be recorded here.
type: string, null
projectId:
description: If the record creation is associated with a project, the project ID should be recorded here.
type: integer, null
purposeIds:
description: If the action being taken by the user involves data and is happening for a specific person, the purpose IDs should be recorded here.
type: array[integer], null
success:
description: Denotes whether the action being audited was successful.
type: boolean
failureReason:
description: Describes the reason that this audit event failed. Possible values are "systemError"
, "insufficientAuthorizations"
, "insufficientPermissions"
, and "userError"
.
type: string
failureDetails:
description: If the audit event failed, details can be provided in this free text field to examine later.
type: string or object
recordType:
description: The type of audit event being captured. This also corresponds to the additional information in the record field. Possible values are "auditQuery"
, "blobVisibility"
, "blobFetch"
, "blobIndex"
, "blobDelete"
, "blobCatalogFetch"
, "blobCatalogFetchDate"
, "blobUpdateFeatures"
, "blobUpdateTags"
, "createQuery"
, "modifyQuery"
, "consoleDataSourceView"
, "sqlAccess"
, "sqlCreateUser"
, "sqlDeleteUser"
, "sqlResetPassword"
, "featureList"
, "sqlQuery"
, "dataSourceCreate"
, "dataSourceDelete"
, "dataSourceSave"
, "dataSourceGet"
, "dataSourceListMine"
, "dataSourceGetTags"
, "dataSourceSubscription"
, "dataSourceGetUsers"
, "dataSourceTest"
, "dictionaryCreate"
, "dictionaryDelete"
, "dictionaryUpdate"
, "projectCreate"
, "projectUpdate"
, "projectDelete"
, "addToProject"
, "removeFromProject"
, "acknowledgePurposes"
, "comment"
, "userVisibilities"
, "accessUser"
, "accessGroup"
, "searchAuthorizations"
, "apiKey"
, "scriptCopy"
, "scriptSave"
, "scriptGet"
, "scriptGetForks"
, "scriptGetVersions"
, "scriptVersionGet"
, "scriptUpdate"
, "scriptDelete"
, "scriptVersionDelete"
,"scriptVersionUpdate"
, "scriptDataSourceGet"
, "scriptDataSourceUpdate"
, "scriptSaveContent"
, "scriptGetContent"
, "userKernelCreate"
, "userKernelUpdate"
, "userKernelDelete"
, "querySampleData"
, "authenticate"
, "checkPendingRequest"
, "policyExemption"
, "governanceUpdate"
, "purposeCreate"
, "purposeUpdate"
, and "purposeDelete"
.
type: string
record:
description: The component-defined type of record. For example, it could be something like 'data source access request'.
type: object
extra:
description: A JSON object representing the additional information to be logged/audited.
type: object
API Key Object
keyIamId:
description: The IAM ID for the user who owns the API key accessed.
type: string
keyId:
description: The API key ID.
type: integer
keyUserId:
description: The user who owns the API key accessed.
type: string
keyAction:
description: Denotes how the specified user was accessed. Possible values are "get"
and "delete"
.
type: string
Data Access Object
accessType:
description: Indicates whether access was granted to an individual blob or if this was a query potentially encompassing many blobs. Possible values are "blob"
and "query"
.
type: string
blobId:
description: If accessType==blob, this is the blobId.
visibility:
description: If the accessType==blob, this is the visibility. If the accessType==query, this is an array of the visibilities the user had when querying.
type: object, array
query:
description: If the accessType==blob, this is not present. If the accessType==query, this is the query.
type: string