Skip to content

Log Aggregation

Audience: Users with the AUDIT permission and Data Owners

Content Summary: Immuta has an advanced logging and auditing system that allows users to easily analyze the system's logs with the most popular log analysis tools.

This page outlines the common message types and JSON properties of audit logs in Immuta.

Log Aggregation for Docker-Based Installations

When running Immuta on Docker-based installations, all logs from the Immuta Docker containers will be sent to the Docker log driver.

The Immuta log file will contain messages that are one-line JSON, as described in Log Formats.

Log Aggregation for RPM-Based Installations

When you are not running Immuta on RPM-based installations, all logs are saved to disk for each web service server. The default log location for the log times are /var/log/immuta/immuta.log. It is the responsibility of the the server administrator to setup log rotation.

The Immuta log file will contain messages that are one-line JSON, as described in Log Formats.

Log Formats

Logs messages from the Immuta platform typically will be one line JSON and contain all of the common JSON properties. Depending on the message type, more JSON properties may be present. See Common Message Types for more details.

Common JSON Properties

  • level: This is a string representation of the log level. Acceptable values are "debug", "info", "warning", "error", and "audit".
  • timestamp: This is a timestamp for when the message occurred. The timestamp format is YYYY-MM-DDTHH:mm:ss.sssZ (ISO 8601).
  • message: This is the log message, which may be used to determine common message types.

Common Message Types

Audit Messages

Each audit message from the Immuta platform will be a one-line JSON object containing the common JSON properties and the Audit JSON properties. Depending on the recordType, an audit message may contain additional data.

In order to discover audit messages using your analysis tool, you may search the object using the criteria below:

  • level: "audit"
  • message: "Audit - *

Audit JSON Properties

  • dateTime:
    • description: The timestamp for when the record was created. This may be an ISO-8601 timestamp string or a ms since epoch timestamp.
    • type: integer or string
    • example: 1504188066580 or "2017-08-31T14:01:15.607Z"
  • component:
    • description: The Immuta component that generated this record. Possible values are "console", "featureStore", "dataSource", "bim", "audit", "script", "policy", "project", "plugin", and "governance".
    • type: string
  • instanceId:
    • description: The instance ID of the component generating this record.
    • type: string
  • profileId:
    • description: The profile ID of the user generating the action.
    • type: integer
  • userId:
    • description: The user ID of the user generating the action.
    • type: string, null
  • sqlUser:
    • description: The database account generating the action.
    • type: string
  • authorizations:
    • description: The user's authorizations at the time of the audited event. This information should be recorded for any type of data retrieval events.
    • type: object
  • dataAccess:
    • description: Describes access to an individual blob or a query that may grant access to multiple blobs.
    • type: object
  • sessionId:
    • description: If this record is generated in response to a user action and if that user's session ID is known, record that session ID here.
    • type : string
  • dataSource:
    • description: If the record creation is associated with a data source, the data source name should be recorded here.
    • type: string, null
  • dataSourceId:
    • description: If the record creation is associated with a data source, the data source ID should be recorded here.
    • type: integer, null
  • projectName:
    • description: If the record creation is associated with a project, the project name should be recorded here.
    • type: string, null
  • projectId:
    • description: If the record creation is associated with a project, the project ID should be recorded here.
    • type: integer, null
  • purposeIds:
    • description: If the action being taken by the user involves data and is happening for a specific person, the purpose IDs should be recorded here.
    • type: array[integer], null
  • success:
    • description: Denotes whether the action being audited was successful.
    • type: boolean
  • failureReason:
    • description: Describes the reason that this audit event failed. Possible values are "systemError", "insufficientAuthorizations", "insufficientPermissions", and "userError".
    • type: string
  • failureDetails:
    • description: If the audit event failed, details can be provided in this free text field to examine later.
    • type: string or object
  • recordType:
    • description: The type of audit event being captured. This also corresponds to the additional information in the record field. Possible values are "auditQuery", "blobVisibility", "blobFetch", "blobIndex", "blobDelete", "blobCatalogFetch", "blobCatalogFetchDate", "blobUpdateFeatures", "blobUpdateTags", "createQuery", "modifyQuery", "consoleDataSourceView", "sqlAccess", "sqlCreateUser", "sqlDeleteUser", "sqlResetPassword", "featureList", "sqlQuery", "dataSourceCreate", "dataSourceDelete", "dataSourceSave", "dataSourceGet", "dataSourceListMine", "dataSourceGetTags", "dataSourceSubscription", "dataSourceGetUsers", "dataSourceTest", "dictionaryCreate", "dictionaryDelete", "dictionaryUpdate", "projectCreate", "projectUpdate", "projectDelete", "addToProject", "removeFromProject", "acknowledgePurposes", "comment", "userVisibilities", "accessUser", "accessGroup", "searchAuthorizations", "apiKey", "scriptCopy", "scriptSave", "scriptGet", "scriptGetForks", "scriptGetVersions", "scriptVersionGet", "scriptUpdate", "scriptDelete", "scriptVersionDelete","scriptVersionUpdate", "scriptDataSourceGet", "scriptDataSourceUpdate", "scriptSaveContent", "scriptGetContent", "userKernelCreate", "userKernelUpdate", "userKernelDelete", "querySampleData", "authenticate", "checkPendingRequest", "policyExemption", "governanceUpdate", "purposeCreate", "purposeUpdate", and "purposeDelete".
    • type: string
  • record:
    • description: The component-defined type of record. For example, it could be something like 'data source access request'.
    • type: object
  • extra:
    • description: A JSON object representing the additional information to be logged/audited.
    • type: object
API Key Object
  • keyIamId:
    • description: The IAM ID for the user who owns the API key accessed.
    • type: string
  • keyId:
    • description: The API key ID.
    • type: integer
  • keyUserId:
    • description: The user who owns the API key accessed.
    • type: string
  • keyAction:
    • description: Denotes how the specified user was accessed. Possible values are "get" and "delete".
    • type: string
Data Access Object
  • accessType:
    • description: Indicates whether access was granted to an individual blob or if this was a query potentially encompassing many blobs. Possible values are "blob"and "query".
    • type: string
  • blobId:
    • description: If accessType==blob, this is the blobId.
  • visibility:
    • description: If the accessType==blob, this is the visibility. If the accessType==query, this is an array of the visibilities the user had when querying.
    • type: object, array
  • query:
    • description: If the accessType==blob, this is not present. If the accessType==query, this is the query.
    • type: string
  • dataSourceTableName:
    • description: The data source table name queried in the audit record.
    • type: string
Blob Fetch Object
  • blobSize:
    • description: The size (in bytes) of the blob being fetched.
    • type: integer
Blob Visibility Object
  • newVisibility:
    • description: This is the new visibility for the blob.
    • type: object
SQL Access Object
  • action:
    • description: Denotes whether access was granted or revoked. Possible values are "revoked"and "granted".
    • type: string
  • sqlUser:
    • description: The username of the user whose access is being manipulated.
    • type: string
SQL Create User Object
  • sqlUser:
    • description: The username of the user whose access is being manipulated.
    • type: string
Data Source Subscription Object
  • dataSourceSubscriptionState:
    • description: If the record type is dataSourceSubscription, this field must be present and indicate the state to which the record was changed (dataSources.length must be 1 in this case). Possible values are "denied", "subscribed", "expert", "owner", "ingest", and "unsubscribed".
    • type: string
  • accessedId:
    • description: The user identifier of the user who is being acted upon.
    • type: integer
  • accessedIdType:
    • description: Type of user being acted upon. Possible values are "user" and "group".
    • type: string
Comment Object
  • commentAction:
    • description: Describes the commenting action. Possible values are "reply", "create", "delete", and "resolve".
    • type: string
  • commentId:
    • description: The comment ID of the comment created / deleted.
    • type: integer
  • commentParentId:
    • description: The parent comment of the comment, if any.
    • type: integer
Data Source Delete Object
  • hardDelete:
    • description: Denotes whether this was a hard delete.
    • type: boolean
Access User Object
  • accessedUserId:
    • description: The user being accessed.
    • type: string
  • accessedIamId:
    • description: The IAM ID for the user being accessed.
    • type: string
  • accessType:
    • description: Denotes how the specified user was accessed. Possible values are "update", "get", "search", "create", "delete", "complete", "newToken", and "clone".
    • type: string
Access Group Object
  • accessedUserId:
    • description: The user being accessed.
    • type: string
  • accessedGroupId:
    • description: The group being accessed.
    • type: integer
  • groupAccessType:
    • description: Denotes how the specified group was accessed. Possible values are "update", "get", "search", "create", "delete", "addUser", and "removeUser".
    • type: string
  • groupIamId:
    • description: The IAM ID for the group being accessed.
    • type: string

Error Response Messages

Each error message response from the Immuta platform will be a one-line JSON object containing the common JSON properties and the error JSON properties below. Error message responses represent error responses that have been sent to clients.

NOTE: It is possible for a similar message referencing the same error to appear in the logs, as this log message represents only that an error response was sent to a client.

In order to discover error response message using your analysis tool, search the object using the criteria below:

  • message: Error Response Sent

Error JSON Properties

  • id:
    • description: A unique ID assigned to each request.
    • type: string
  • method:
    • The HTTP method used for the request.
    • type: string
  • path:
    • description: This is the HTTP path used for the request.
    • type: string
  • query:
    • description: An object containing the parsed query string used for the query.
    • type: object
  • responseSentTime:
    • description: This is the duration from the time a request was received until the time that the server responded.
    • type: integer
  • responseTime:
    • description: This is the duration from the time a request was received until server post-processing.
    • type: integer
  • stack:
    • description: This is a string representation of a stack trace if one exists.
    • type: string
  • statusCode:
    • description: This is the HTTP status code that was sent to the client.
    • type: integer

Request Response Messages

Each request message response from the Immuta platform will be a one-line JSON object containing the common JSON properties and the error JSON properties below. Request message responses represent responses that have been sent to clients.

In order to discover request response messages using your analysis tool, search the object using the criteria below:

  • message: Response Sent

Request JSON Properties

  • id:
    • description: A unique ID assigned to each request.
    • type: string
  • method:
    • The HTTP method used for the request.
    • type: string
  • path:
    • description: This is the HTTP path used for the request.
    • type: string
  • query:
    • description: An object containing the parsed query string used for the query.
    • type: object
  • responseSentTime:
    • description: This is the duration from the time a request was received until the time that the server responded.
    • type: integer
  • responseTime:
    • description: This is the duration from the time a request was received until server post-processing.
    • type: integer
  • statusCode:
    • description: This is the HTTP status code that was sent to the client.
    • type: integer