Apache.Arrow.Adbc.Drivers.Databricks 0.22.0

Databricks Driver

The Databricks ADBC driver is built on top of the Spark ADBC driver and inherits all of its properties, plus additional Databricks-specific functionality.

Database and Connection Properties

Note: The Databricks driver inherits all properties from the Spark driver. The properties below are Databricks-specific additions.

Configuration Methods

The Databricks driver supports multiple ways to configure properties:

1. Direct Property Configuration

Pass properties directly when creating the driver connection (traditional method).

2. Environment Variable Configuration

Configure properties using a JSON file loaded via environment variables:

1. Create a JSON configuration file with standard ADBC parameters:

{
  "adbc.databricks.driver_config_take_precedence": "true",
  "adbc.databricks.enable_pk_fk": "false",
  "adbc.connection.catalog": "my_catalog",
  "adbc.connection.db_schema": "my_schema"
}

Note

All values in the JSON configuration file must be strings (including numbers, booleans, and file paths). For example, use "true" instead of true, and "4443" instead of 4443.

Example: Using mitmproxy to Inspect Thrift Traffic

To inspect Thrift traffic using mitmproxy, you can configure the Databricks driver to use a local proxy with TLS interception. Below is an example JSON configuration:

{
  "adbc.databricks.driver_config_take_precedence": "true",
  "adbc.proxy_options.use_proxy" : "true",
  "adbc.proxy_options.proxy_host" : "localhost",
  "adbc.proxy_options.proxy_port" : "4443",
  "adbc.http_options.tls.enabled": "true",
  "adbc.http_options.tls.allow_self_signed" : "true",
  "adbc.http_options.tls.disable_server_certificate_validation" : "true",
  "adbc.http_options.tls.allow_hostname_mismatch" : "true",
  "adbc.http_options.tls.trusted_certificate_path" : "C:\\your-path-to\\mitmproxy-ca-cert.pem"
}

2. Set the system environment variable DATABRICKS_CONFIG_FILE to point to your JSON file:

   1. Open System Properties → Advanced → Environment Variables
   2. Add new system variable: Name=DATABRICKS_CONFIG_FILE, Value=C:\path\to\your\config.json

3. Property Merging Behavior:

   • By default: Constructor/code properties override environment config properties
   • With "adbc.databricks.driver_config_take_precedence": "true": Environment config properties override constructor/code properties

3. Hybrid Configuration

You can combine both methods - the driver will automatically merge environment config with constructor properties based on the precedence setting.

Use Cases:

• PowerBI Integration: Set system-wide defaults via environment config while allowing connection-specific overrides

Authentication Properties

CloudFetch Properties

CloudFetch is Databricks' high-performance result retrieval system that downloads result data directly from cloud storage.

Databricks-Specific Properties

Tracing Properties

Authentication Methods

The Databricks ADBC driver supports the following authentication methods:

1. Token-based Authentication

Using a Databricks personal access token:

• Set adbc.spark.auth_type to oauth
• Set adbc.databricks.oauth.grant_type to access_token (this is the default if not specified)
• Set adbc.spark.oauth.access_token to your Databricks personal access token

2. OAuth Client Credentials Flow

For machine-to-machine (m2m) authentication:

• Set adbc.spark.auth_type to oauth
• Set adbc.databricks.oauth.grant_type to client_credentials
• Set adbc.databricks.oauth.client_id to your OAuth client ID
• Set adbc.databricks.oauth.client_secret to your OAuth client secret
• Set adbc.databricks.oauth.scope to your auth scope (defaults to sql)

The driver will automatically handle token acquisition, renewal, and authentication with the Databricks service.

Note: Basic (username and password) authentication is not supported at this time.

Server-Side Properties

Server-side properties allow you to configure Databricks session settings. Any property with the adbc.databricks.ssp_ prefix will be passed to the server by executing SET key=value queries.

For example, setting adbc.databricks.ssp_use_cached_result to true will result in executing SET use_cached_result=true on the server when the session is opened.

The property name after the ssp_ prefix becomes the server-side setting name.

Data Types

The following table depicts how the Databricks ADBC driver converts a Databricks type to an Arrow type and a .NET type:

Tracing

Tracing Exporters

To enable tracing messages to be observed, a tracing exporter needs to be activated. Use either the environment variable OTEL_TRACES_EXPORTER or the parameter adbc.traces.exporter to select one of the supported exporters. The parameter has precedence over the environment variable. The parameter must be set before the connection is initialized.

The following exporters are supported:

File Exporter (adbcfile)

Rotating trace files are written to a folder. The file names are created with the following pattern: apache.arrow.adbc.drivers.bigquery-<YYYY-MM-DD-HH-mm-ss-fff>-<process-id>.log.

The folder used depends on the platform.

By default, up to 999 files of maximum size 1024 KB are written to the trace folder.