Databricks Install & Auth

Overview

Set up Databricks CLI and SDK with authentication credentials.

Prerequisites

• Python 3.8+ with pip
• Databricks workspace URL (e.g., https://adb-1234567890.1.azuredatabricks.net)
• Personal Access Token or OAuth credentials
• Admin access for Unity Catalog setup

Instructions

Step 1: Install Databricks CLI and SDK

set -euo pipefail
# Install CLI v2 (recommended)
pip install databricks-cli

# Install Python SDK
pip install databricks-sdk

# Install dbx for local development
pip install dbx

Step 2: Configure Authentication

Option A: Personal Access Token (Simplest)

# Interactive configuration
databricks configure --token

# Enter workspace URL: https://adb-1234567890.1.azuredatabricks.net
# Enter token: dapi1234567890abcdef...

Option B: Environment Variables

# Set environment variables
export DATABRICKS_HOST="https://adb-1234567890.1.azuredatabricks.net"
export DATABRICKS_TOKEN="dapi1234567890abcdef..."

# Or create .env file
cat > .env << 'EOF'
DATABRICKS_HOST=https://adb-1234567890.1.azuredatabricks.net
DATABRICKS_TOKEN=dapi1234567890abcdef...
EOF

Option C: OAuth (Recommended for Production)

# Using OAuth U2M (user-to-machine)
databricks configure --oauth

# Using OAuth M2M (service principal)
export DATABRICKS_HOST="https://adb-1234567890.1.azuredatabricks.net"
export DATABRICKS_CLIENT_ID="your-client-id"
export DATABRICKS_CLIENT_SECRET="your-client-secret"

Step 3: Configure Profiles (Multi-Workspace)

# ~/.databrickscfg
[DEFAULT]
host = https://adb-dev-1234567890.1.azuredatabricks.net
token = dapi_dev_token...

[staging]
host = https://adb-staging-1234567890.1.azuredatabricks.net
token = dapi_staging_token...

[production]
host = https://adb-prod-1234567890.1.azuredatabricks.net
token = dapi_prod_token...

Step 4: Verify Connection

# Test CLI connection
databricks workspace list /

# Test with specific profile
databricks workspace list / --profile staging

# Test SDK connection
python -c "
from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
print(f'Connected to: {w.config.host}')
print(f'Clusters: {len(list(w.clusters.list()))}')
"

Output

• Databricks CLI installed and configured
• Python SDK installed
• Authentication credentials stored
• Connection verified to workspace

Error Handling

Examples

Python SDK Initialization

from databricks.sdk import WorkspaceClient

# Auto-detect from environment/config file
w = WorkspaceClient()

# Explicit configuration
w = WorkspaceClient(
    host="https://adb-1234567890.1.azuredatabricks.net",
    token="dapi1234567890abcdef..."
)

# With profile
w = WorkspaceClient(profile="production")

Service Principal Authentication

from databricks.sdk import WorkspaceClient
from databricks.sdk.config import Config

config = Config(
    host="https://adb-1234567890.1.azuredatabricks.net",
    client_id="your-client-id",
    client_secret="your-client-secret"
)
w = WorkspaceClient(config=config)

Azure AD Authentication

from databricks.sdk import WorkspaceClient
from azure.identity import DefaultAzureCredential

# Uses Azure AD managed identity
w = WorkspaceClient(
    host="https://adb-1234567890.1.azuredatabricks.net",
    credentials_provider=DefaultAzureCredential()
)

Resources

• Databricks CLI Documentation
• Databricks SDK for Python
• Authentication Methods
• Personal Access Tokens

Next Steps

After successful auth, proceed to databricks-hello-world for your first cluster and notebook.