HintsToday

Here’s a complete guide to architecting and implementing data governance using Unity Catalog on Databricks — the unified governance layer designed to manage access, lineage, compliance, and auditing across all workspaces and data assets.

✅ Why Unity Catalog for Governance?

Unity Catalog offers:

🏗️ Unity Catalog Governance Framework – Architecture

🔸 Architecture Overview:

                 +---------------------+
                 |      Users/Roles    |
                 +----------+----------+
                            |
                     [Access Policies]
                            ↓
          +-----------------+------------------+
          |                                    |
     Unity Catalog                         Audit Logs
  (Access + Lineage)                      (Compliance)
          |
   +------+-------+-------+-------+
   |      |       |       |       |
Catalogs Schemas Tables Views Tags

🔧 Governance Setup Components

✅ Step-by-Step: Implement Data Governance Framework

🔹 Step 1: Enable Unity Catalog (One-Time Setup)

1. Enable Unity Catalog from your Azure Databricks admin console.
2. Set up metastore (shared governance layer across workspaces).
3. Assign Metastore Admins and Workspace Bindings.

🔹 Step 2: Define Catalog Hierarchy

Organize your data assets like this:

sales_catalog
├── raw_schema
│   └── sales_raw
├── clean_schema
│   └── sales_clean
└── curated_schema
    └── sales_summary

Create them:

CREATE CATALOG IF NOT EXISTS sales_catalog;
CREATE SCHEMA IF NOT EXISTS sales_catalog.clean_schema;

🔹 Step 3: Apply Fine-Grained Access Control

🔸 Example 1: Table-Level Permissions

GRANT SELECT ON TABLE sales_catalog.clean_schema.sales_clean TO `data_analyst_group`;
GRANT ALL PRIVILEGES ON TABLE sales_catalog.curated_schema.sales_summary TO `finance_admins`;

🔸 Example 2: Column Masking for PII

CREATE MASKING POLICY mask_email
AS (email STRING) RETURNS STRING ->
  CASE
    WHEN is_account_group_member('pii_viewers') THEN email
    ELSE '*** MASKED ***'
  END;

ALTER TABLE sales_catalog.customer_schema.customers
  ALTER COLUMN email
  SET MASKING POLICY mask_email;

🔹 Step 4: Add Tags for Classification

ALTER TABLE sales_catalog.customer_schema.customers
  ALTER COLUMN email
  SET TAGS ('classification' = 'pii', 'sensitivity' = 'high');

ALTER SCHEMA customer_schema SET TAGS ('owner' = 'data-steward@company.com');

🔹 Step 5: Enable and Use Data Lineage

Once enabled, Unity Catalog automatically tracks full lineage:

• View lineage in Data Explorer
• Shows source → intermediate → gold table/view
• Tracks jobs, notebooks, queries

🔹 Step 6: Monitor & Audit Usage

Enable audit logging:

• At the workspace and Unity Catalog level
• Export logs to Azure Log Analytics, Storage, or SIEM

Includes:

• Query execution history
• Access violations
• Data masking attempts

🧰 Optional Add-ons for Governance

🧪 Example: Row-Level Security (Dynamic View)

CREATE OR REPLACE VIEW sales_catalog.curated_schema.secure_sales_summary AS
SELECT * FROM sales_summary
WHERE region = current_user_region();

Where current_user_region() is a UDF tied to user metadata.

🧾 Best Practices

✅ Summary Framework Checklist