Skip to content

Create, Apply, and Clone a Framework

Note

In previous documentation, rule is referred to as classifier or identifier and framework is referred to as template.

Create a framework

  1. Generate your API key on the API Keys tab on your profile page and save the API key somewhere secure. You will include this API key in the authorization header when you make a request to the Immuta API.

  2. Find rules to include in your framework using one of these methods:

    Immuta CLI

    immuta api sdd/classifier?sortField=name&sortOrder=asc&limit=25&searchText=RULE

    HTTP API

    curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: 12345678900000" \
    https://your-immuta-url.immuta.com/sdd/classifier?sortField=name&sortOrder=asc&limit=25&searchText=RULE

  3. If the request was successful, you will receive a list of available rules.

    {
  "count": 3,
  "hits": [
    {
      "createdBy": {
        "id": 1,
        "name": "John",
        "email": "john@example.com"
      },
      "name": "ACCOUNT_NUMBER_RULE",
      "displayName": "Account Number Rule",
      "description": "This rule recognizes account numbers using a regex",
      "type": "regex",
      "config": {
        "tags": [
          "Discovered.account-number"
        ],
        "regex": "[0-9]{9}-[0-9]{3}-[0-9]{1}",
        "minConfidence": 0.5
      },
      "id": 104,
      "createdAt": "2021-10-20T19:12:24.889Z",
      "updatedAt": "2021-10-20T19:12:24.889Z"
    },
    {
      "createdBy": {
        "id": 1,
        "name": "John",
        "email": "john@example.com"
      },
      "name": "EMPLOYEE_DESK_LOCATION_RULE",
      "displayName": "Employee Desk Location Rule",
      "description": "This rule recognizes an employee's desk location.",
      "type": "dictionary",
      "config": {
        "tags": [
          "Discovered.desk-location"
        ],
        "values": [
          "Research Lab",
          "Blue Room",
          "Purple Room"
        ],
        "caseSensitive": false,
        "minConfidence": 0.6
      },
      "id": 68,
      "createdAt": "2021-10-20T17:57:51.696Z",
      "updatedAt": "2021-10-20T17:57:51.696Z"
    },
    {
      "createdBy": {
        "id": 1,
        "name": "John",
        "email": "john@example.com"
      },
      "name": "SOCIAL_SECURITY_NUMBER_COLUMNS_RULE",
      "displayName": "Social Security Number Columns Rule",
      "description": "This Rule recognizes column names that match the defined regex pattern.",
      "type": "columnNameRegex",
      "config": {
        "tags": [
          "Discovered.Social Security Numbers"
        ],
        "columnNameRegex": "ssn|social ?security"
      },
      "id": 67,
      "createdAt": "2021-10-20T17:57:17.930Z",
      "updatedAt": "2021-10-20T17:57:17.930Z"
    }
  ]
}

  4. Save the framework payload in a .json file. Use the tabs below to see different examples of frameworks.

    {
  "name": "ACCOUNT_NUMBERS_FRAMEWORK",
  "displayName": "Account Numbers Framework",
  "description": "This framework contains the rule that recognizes account numbers.",
  "classifiers": [
    {
      "name": "ACCOUNT_NUMBER_RULE"
    }
  ],
  "sampleSize": 100
}

    {
  "name": "EMPLOYEE_DESK_LOCATION_FRAMEWORK",
  "displayName": "Employee Desk Location Framework",
  "description": "This framework contains the rule that recognizes the name of the room an employee's desk is in.",
  "classifiers": [
    {
      "name": "EMPLOYEE_DESK_LOCATION_RULE"
    }
  ],
  "sampleSize": 100
}

    {
  "name": "SOCIAL_SECURITY_NUMBERS_FRAMEWORK",
  "displayName": "Social Security Numbers Framework",
  "description": "This framework contains the rule that matches social security number column names with the defined regex.",
  "classifiers": [
    {
      "name": "SOCIAL_SECURITY_NUMBER_COLUMNS_RULE"
    }
  ],
  "sampleSize": 100
}

    {
  "name": "STUDENT_LOCATION_FRAMEWORK",
  "displayName": "Student Location Framework",
  "description": "This framework contains the rule that recognizes student location information such as, a student's residence hall, floor, or room.",
  "classifiers": [
    {
      "name": "STUDENT_LOCATION_RULE"
    }
  ],
  "sampleSize": 100
}

  5. Create the framework:

    Immuta CLI

    immuta api sdd/template -X POST --input ./example-payload.json

    HTTP API

    curl \
    --request POST \
    --header "Content-Type: application/json" \
    --header "Authorization: 12345678900000" \
    --data @example-payload.json \
    https://your-immuta-url.immuta.com/sdd/template

  6. If the request is successful, you will receive a response that contains details about the framework. Use the tabs below to see different responses for different frameworks.

    {
  "name": "ACCOUNT_NUMBERS_FRAMEWORK",
  "displayName": "Account Numbers Framework",
  "description": "This framework contains the rule that recognizes account numbers.",
  "sampleSize": 100,
  "createdBy": {
    "id": 1,
    "name": "John",
    "email": "john@example.com"
  },
  "id": 1,
  "createdAt": "2021-10-21T19:12:22.092Z",
  "updatedAt": "2021-10-21T19:12:22.092Z",
  "classifiers": [
    {
      "name": "ACCOUNT_NUMBER_RULE",
      "overrides": {}
    }
  ]
}

    After the framework is applied to data sources and sensitive data discovery is run, the Discovered.account-number tag will be applied to columns that Immuta identifies with 50% confidence, as configured in the rule.

    {
  "name": "EMPLOYEE_DESK_LOCATION_FRAMEWORK",
  "displayName": "Employee Desk Location Framework",
  "description": "This framework contains the rule that recognizes employee location information.",
  "sampleSize": 100,
  "createdBy": {
    "id": 1,
    "name": "John",
    "email": "john@example.com"
  },
  "id": 1,
  "createdAt": "2021-10-21T18:03:58.967Z",
  "updatedAt": "2021-10-21T18:03:58.967Z",
  "classifiers": [{
    "name": "EMPLOYEE_DESK_LOCATION_RULE",
    "overrides": {}
  }]
}

    After the framework is applied to data sources and sensitive data discovery is run, the Discovered.desk-location tag will be applied to columns when Immuta recognizes the values Research Lab, Blue Room, or Purple Room with 60% confidence, as configured in the rule.

    {
  "name": "SOCIAL_SECURITY_NUMBERS_FRAMEWORK",
  "displayName": "Social Security Numbers Framework",
  "description": "This framework contains the rule that matches social security number column names with the defined regex.",
  "sampleSize": 100,
  "createdBy": {
    "id": 1,
    "name": "John",
    "email": "john@example.com"
  },
  "id": 2,
  "createdAt": "2021-10-21T19:12:22.092Z",
  "updatedAt": "2021-10-21T19:12:22.092Z",
  "classifiers": [
    {
      "name": "SOCIAL_SECURITY_NUMBER_COLUMNS_RULE",
      "overrides": {}
    }
  ]
}

    After the framework is applied to data sources and sensitive data discovery is run, the Discovered.social-security-number tag will be applied to columns that have a name that match the ssn|social ?security regex, such as ssn, socialsecurity, or social security.

    {
  "name": "STUDENT_LOCATION_FRAMEWORK",
  "displayName": "Student Location Framework",
  "description": "This framework contains the rule that recognizes when a student's residence hall, floor, or room appears in a dataset.",
  "sampleSize": 100,
  "createdBy": {
    "id": 1,
    "name": "John",
    "email": "john@example.com"
  },
  "id": 1,
  "createdAt": "2021-10-21T18:03:58.967Z",
  "updatedAt": "2021-10-21T18:03:58.967Z",
  "classifiers": [{
    "name": "STUDENT_LOCATION_RULE",
    "overrides": {}
  }]
}

    After the framework is applied to data sources and sensitive data discovery is run, the Discovered.residence-hall tag will be applied to columns when Immuta recognizes values that match those listed in the Residence Halls data source with 70% confidence, as configured in the rule.

Apply a framework to data sources

All the attributes for rules and frameworks are provided on the Sensitive data discovery API page. However, attributes specific to this section are outlined in the table below.

Attribute Description
template string The name of the framework to apply to the data sources; null clears the current framework.
sources string The name of the data sources to apply the framework to.

  1. Find frameworks to apply to your data sources:

    Immuta CLI

    immuta api sdd/template

    HTTP API

    curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: 12345678900000" \
    https://your-immuta-url.immuta.com/sdd/template

  2. If the request was successful, you will receive a list of available frameworks.

    {
  "count": 3,
  "hits": [
    {
      "name": "ACCOUNT_NUMBERS_FRAMEWORK",
      "displayName": "Account Numbers Framework",
      "description": "This framework contains the rule that recognizes account numbers.",
      "sampleSize": 100,
      "createdBy": {
        "id": 1,
        "name": "John",
        "email": "john@example.com"
      },
      "id": 2,
      "createdAt": "2021-10-20T19:13:35.319Z",
      "updatedAt": "2021-10-20T19:13:35.319Z",
      "classifiers": [
        {
          "name": "ACCOUNT_NUMBER_RULE",
          "overrides": {}
        }
      ]
    },
    {
      "name": "EMPLOYEE_DESK_LOCATION_FRAMEWORK",
      "displayName": "Employee Desk Location Framework",
      "description": "Contains a rule that recognizes the name of a room a desk is in.",
      "sampleSize": 100,
      "createdBy": {
        "id": 1,
        "name": "John",
        "email": "john@example.com"
      },
      "id": 1,
      "createdAt": "2021-10-20T18:03:58.967Z",
      "updatedAt": "2021-10-20T18:03:58.967Z",
      "classifiers": [
        {
          "name": "EMPLOYEE_DESK_LOCATION_RULE",
          "overrides": {}
        }
      ]
    },
    {
      "name": "SOCIAL_SECURITY_NUMBERS_FRAMEWORK",
      "displayName": "Social Security Numbers Framework",
      "description": "Contains a rule that matches ssn column names with the defined regex.",
      "sampleSize": 100,
      "createdBy": {
        "id": 1,
        "name": "John",
        "email": "john@example.com"
      },
      "id": 3,
      "createdAt": "2021-10-20T19:13:58.359Z",
      "updatedAt": "2021-10-20T19:13:58.359Z",
      "classifiers": [
        {
          "name": "SOCIAL_SECURITY_NUMBER_COLUMNS_RULE",
          "overrides": {}
        }
      ]
    }
  ]
}

  3. Select an appropriate framework to apply to your data sources, and save the payload in a .json file:

    {
  "template": "ACCOUNT_NUMBERS_FRAMEWORK",
  "sources": [
    "Insurance Data"
  ]
}

  4. Apply the framework to your data sources:

    Immuta CLI

    immuta api sdd/template/apply -X PUT --input ./example-payload.json

    HTTP API

    curl \
    --request PUT \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    --data @example-payload.json \
    https://your-immuta-url.immuta.com/sdd/template/apply

  5. You will receive a response that indicates whether or not the framework was successfully applied to your data sources.

    {
  "success": true
}

Clone a framework

Users cannot modify frameworks created by other data owners, but they can clone frameworks and make changes to the clone.

  1. Get a list of frameworks to determine the framework you want to clone using one of these methods:

    Immuta CLI

    immuta api sdd/sdd/template?sortField=name&sortOrder=asc&offset=5&limit=5

    HTTP API

    curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: 12345678900000" \
    https://your-immuta-url.immuta.com/sdd/template?sortField=name&sortOrder=asc&offset=5&limit=5

  2. Save the framework clone name and details in a .json file.

    {
  "name": "INSURANCE_ACCOUNT_NUMBERS",
  "displayName": "Insurance Account Numbers",
  "description": "This framework is specific to insurance accounts."
}

  3. Clone the framework:

    Immuta CLI

    immuta api sdd/template/ACCOUNT_NUMBERS_FRAMEWORK/clone -X POST --input ./example-payload.json

    HTTP API

    curl \
    --request POST \
    --header "Content-Type: application/json" \
    --header "Authorization: 12345678900000" \
    --data @example-payload.json \
    https://your-immuta-url.immuta.com/sdd/template/ACCOUNT_NUMBERS_FRAMEWORK/clone

  4. If the request was successful, you will receive a response that provides details about the new framework.

    {
  "name": "INSURANCE_ACCOUNT_NUMBERS",
  "displayName": "Insurance Account Numbers",
  "description": "This framework is specific to insurance accounts.",
  "sampleSize": 100,
  "createdBy": {
    "id": 1,
    "name": "John",
    "email": "john@example.com"
  },
  "id": 4,
  "createdAt": "2021-10-20T20:48:37.805Z",
  "updatedAt": "2021-10-20T20:48:37.805Z",
  "classifiers": [
    {
      "name": "ACCOUNT_NUMBER_RULE",
      "overrides": {}
    }
  ]
}

You can now modify the framework, such as changing the rules or sample size.

Configure resulting tags and confidence

minConfidence and sampleSize are not supported for native SDD

To disable a resulting tag from being set, you can create a framework that overrides the rule that contains that tag.

For example, the built-in PERSON_NAME rule has the following resulting tags: Discovered.PHI, Discovered.PII, Discovered.Entity.Person Name, and Discovered.Identifier Indirect. However, your organization doesn't have any health data, so you don't want the PHI tag to be applied to your data sources, but you do want all the other resulting tags within that rule.

To override the Discovered.PHI tag, you would create a framework that includes the PERSON_NAME rule and removes the Discovered.PHI from the resulting tags in the framework payload.

  1. View the details about the PERSON_NAME rule so you know what to include in your framework using one of these methods:

    Immuta CLI

    immuta api sdd/classifier?sortField=name&sortOrder=asc&limit=25&searchText=PERSON_NAME

    HTTP API

    curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: 12345678900000" \
    https://your-immuta-url.immuta.com/sdd/classifier?sortField=name&sortOrder=asc&limit=25&searchText=PERSON_NAME

  2. If the request was successful, the response will include details about the PERSON_NAME rule.

    {
  "createdBy": {
    "id": 21,
    "name": "Immuta System Account",
    "email": "immuta_system@immuta.com"
  },
  "name": "PERSON_NAME",
  "displayName": "Person Name",
  "description": "Detects strings consistent with a dictionary of people's names.",
  "type": "builtIn",
  "config": {
    "tags": [
      "Discovered.PHI",
      "Discovered.PII",
      "Discovered.Entity.Person Name",
      "Discovered.Identifier Indirect"
    ],
    "minConfidence": 0.3
  },
  "id": 54,
  "createdAt": "2021-10-21T07:35:14.416Z",
  "updatedAt": "2021-10-21T12:57:43.919Z"
}

  3. Remove the Discovered.PHI tag from the resulting tags in the rule config, and save the template payload in a .json file.

    {
  "name": "PERSON_NAME_OVERRIDE",
  "displayName": "Person Name Override",
  "description": "This framework removes the PHI tag from the PERSON_NAME rule.",
  "classifiers": [
    {
      "name": "PERSON_NAME",
        "overrides": {
          "tags": [
            "Discovered.PII",
            "Discovered.Entity.Person Name",
            "Discovered.Identifier Indirect"
          ]
        }
      }
    ],
  "sampleSize": 100
}

  4. Create the framework:

    Immuta CLI

    immuta api sdd/template -X POST --input ./example-payload.json

    HTTP API

    curl \
    --request POST \
    --header "Content-Type: application/json" \
    --header "Authorization: 12345678900000" \
    --data @example-payload.json \
    https://your-immuta-url.immuta.com/sdd/template

If the request is successful, you will receive a response that details the new framework:

{
  "name": "PERSON_NAME_OVERRIDE",
  "displayName": "Person Name Override",
  "description": "This framework removes the PHI tag from the PERSON_NAME rule.",
  "sampleSize": 100,
  "createdBy": {
    "id": 1,
    "name": "John",
    "email": "john@example.com"
  },
  "id": 1,
  "createdAt": "2021-10-21T17:11:18.057Z",
  "updatedAt": "2021-10-21T17:11:18.057Z",
  "classifiers": [
    {
      "name": "PERSON_NAME",
      "overrides": {
        "tags": [
          "Discovered.PII",
          "Discovered.Entity.Person Name",
          "Discovered.Identifier Indirect"
        ]
      }
    }
  ]
}

What's next

Now that you've created a framework, continue to one of the following tutorials: