Skip to content

Create Data Sources API Examples

Audience: Data Engineers

Content Summary: This page contains example request payloads for creating data sources.

Basic Data Source

connectionKey: my-databricks
connection:
    hostname: your.databricks.hostname.com
    port: 443
    ssl: true
    database: tpc
    username: token
    password: "${DATABRICKS_PASSWORD}"
    httpPath: sql/protocolv1/o/0/11101101
    handler: Databricks

Data Source (with More Options)

connectionKey: my-databricks
nameTemplate:
  dataSourceFormat: Databricks <Tablename>
  tableFormat: <tablename>
  schemaFormat: databricks
connection:
  hostname: your.databricks.hostname.com
  port: 443
  ssl: true
  database: data
  username: token
  password: "${DATABRICKS_PASSWORD}"
  httpPath: sql/protocolv1/o/0/1110-11123
  handler: Databricks
sources:
  - table: credit_card_transactions
    schema: data
    tags:
      table:
    - PCI
    - SENSITIVE
    columns
    - columnName: transaction_date
      tags:
      - PCI
      - DATE
  - table: crime_data
    schema: data
    naming:
        datasource: Crime Data
        table: crime_data
        schema: databricks

Databricks Data Source (Override Naming Convention)

connectionKey: ebock-databricks
nameTemplate:
  dataSourceFormat: Databricks <Tablename>
  tableFormat: <tablename>
  schemaFormat: databricks
connection:
  hostname: your.databricks.hostname.com
  port: 443
  ssl: true
  database: ebock
  username: token
  password: "${DATABRICKS_PASSWORD}"
  httpPath: sql/protocolv1/o/0/1110-185737-wove
  handler: Databricks
sources:
  - table: credit_card_transactions
    schema: ebock
  - table: crime_data_delta
    schema: ebock
    naming:
        datasource: Crime Data
        table: crime_data
        schema: databricks
  - table: hipaa_data
    schema: ebock

Impala Data Source (with userFile)

connectionKey: cdh-impala
nameTemplate:
  dataSourceFormat: Impala <Tablename>
  tableFormat: <tablename>
  schemaFormat: impala
connection:
  hostname: your.hadoop.hostname.example.com
  port: 21050
  ssl: true
  database: default
  handler: Apache Impala
  authenticationMethod: kerberos
  username: usera
  userFiles:
    - keyName: TrustedCerts
      content: <Base64 encoded contents of file go here>
      userFilename: tls-ca-bundle.pem
sources:
  - table: medical_records_parquet
    schema: default
  - table: nyc_taxi_fare_parquet
    schema: default
  - table: nyc_taxi_trip_parquet
    schema: default

Basic Postgres Data Source

connectionKey: tpc-postgres
connection:
  hostname: limited.databases.test.example.io
  port: 5432
  ssl: true
  database: tpc
  username: test
  password: "${LIMITED_PASSWORD}"
  schema: public
  handler: PostgreSQL

Snowflake Data Source (Specify Sources)

connectionKey: tpc-snowflake
nameTemplate:
  dataSourceFormat: Snowflake <Tablename>
  tableFormat: <tablename>
  schemaFormat: snowflake
connection:
  hostname: example.hostname.snowflakecomputing.com
  port: 443
  ssl: true
  database: TPC
  username: USERA
  password: "${SNOWFLAKE_PASSWORD}"
  schema: PUBLIC
  warehouse: IT_WH
  handler: Snowflake
sources:
  - table: CASE
    schema: PUBLIC
  - table: CASE2
    schema: PUBLIC
  - table: CUSTOMER
    schema: PUBLIC
  - table: WEB_SALES
    schema: PUBLIC