Databricks Zerobus Ingest Integration Guide
Use this guide to set up an integration between Litmus Edge and Databricks Zerobus Ingest. Once the integration is set up, you can use it to publish data from Litmus Edge directly into Unity Catalog Delta tables using the Zerobus ingestion pipeline.
Note: This connector publishes data from Litmus Edge to Databricks only. Inbound subscriptions are not supported.
Before You Begin
Before configuring the connector in Litmus Edge, complete the following setup steps in Databricks.
See the following Databricks resources to learn more about Zerobus Ingest and configuring your workspace:
Step 1: Create a Cloud Storage Bucket
Create a cloud storage bucket or container appropriate for your Databricks deployment. For example, use an AWS S3 bucket, Azure Blob Storage container, or Google Cloud Storage bucket. This storage will serve as the external location for your Databricks Catalog.
Refer to your cloud provider's documentation for creating a bucket or container and note the bucket or container URI for use in the next step.
Step 2: Create an External Location in Databricks
In your Databricks workspace, create an External Location and link it to the cloud storage bucket created in Step 1.
See Create an external location for detailed instructions.
Step 3: Create a Service Principal and OAuth Secret
Create a Service Principal in your Databricks workspace, then generate an OAuth secret for it. Litmus Edge uses these credentials to authenticate with the Databricks Zerobus Ingest endpoint.
After creation, record the following values. You will need them when configuring the connector in Litmus Edge:
- Client ID The unique identifier for the service principal, in UUID format (for example xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
- OAuth Secret The secret generated for the service principal
Important: Store the OAuth secret securely. It is displayed only once at creation time.
See Manage service principals for detailed instructions.
Step 4: Create a Catalog
In your Databricks workspace, create a new Catalog backed by the external location created in Step 2.
Important: Do not use a catalog whose Storage Location is set to Default Storage. The Zerobus Ingest connector requires a catalog backed by an explicit external location.
See Create catalog for detailed instructions.
Step 5: Create a Schema
In the catalog, create a new Schema or use the automatically created default schema.
See Create schemas for detailed instructions.
Step 6: Create a Table
Create a table in the schema with columns that match the data fields Litmus Edge will publish. The following example creates a table compatible with the Litmus Edge DeviceHub tag payload:
CREATE TABLE IF NOT EXISTS <catalog>.<schema>.<table> (
success BOOLEAN,
datatype STRING,
timestamp BIGINT,
registerId STRING,
value DOUBLE,
deviceID STRING,
tagName STRING,
deviceName STRING,
description STRING,
metadata VARIANT
);Replace <catalog>, <schema>, and <table> with the names you chose in Steps 4 and 5.
For more information on table creation syntax and identifier naming restrictions, see:
Note: This guide uses DeviceHub tag payload as an example for the Databricks table schema and outbound topic configuration. Litmus Edge also supports streaming custom JSON structures produced by Analytics or Digital Twins to Databricks. The steps are the same, but you will need to define your Databricks SQL Table columns to match your custom payload structure.
Step 7: Grant Permissions to the Service Principal
Grant the service principal created in Step 3 the necessary permissions on the catalog and schema. Run the following SQL statements in your Databricks workspace, replacing <catalog>, <schema>, <table>, and <client-id> with your values:
GRANT USE CATALOG ON CATALOG <catalog> TO `<client-id>`;
GRANT USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<client-id>`;
GRANT MODIFY, SELECT ON TABLE <catalog>.<schema>.<table> TO `<client-id>`;See Unity Catalog privileges and securable objects for more information.
Set up the Outbound Connection (Publish to Databricks)
Follow the steps below to configure Litmus Edge to publish data to Databricks.
Step 1: Add the Databricks Zerobus Ingest Connector
Follow the steps to Add a Connector and select the Databricks Zerobus Ingest connector provider.
Configure the following parameters:
- Name: The name of the connector.
- Workspace URL: The URL of your Databricks workspace (for example https://<workspace-id>.cloud.databricks.com).
- Zerobus Ingest URL: The Zerobus ingestion endpoint for your workspace. The format depends on your cloud provider:
- AWS: <workspace-id>.zerobus.<region>.cloud.databricks.com (see Zerobus Ingest on AWS)
- Azure: <workspace-id>.zerobus.<region>.azuredatabricks.net (see Zerobus Ingest on Azure)
- Client ID: The Client ID of the service principal created in the Before You Begin section.
- OAuth Secret: The OAuth secret generated for the service principal.
- Catalog: The name of the Databricks catalog created in the Before You Begin section.
- Schema: The name of the schema within the catalog (for example default).
- Default Table: The name of the target table within the schema.
Step 2: Enable the Connector
After adding the connector, click the toggle in the connector tile to enable it.
If you see a Failed status, review the Manage Connectors page and the relevant error messages.
Step 3: Add an Outbound Topic
After the connector is enabled, add one or more outbound topics to define which Litmus Edge data is published to Databricks.
To add an outbound topic:
- Click the connector tile. The connector Dashboard appears.
- Click the Topics tab.
- Click the Add a new topic icon. The Data Integration dialog box appears.
- Configure the following parameters:
- Data Direction: Select Local to Remote - Outbound.
- Local Data Topic: Select the DeviceHub tag or topic from Litmus Edge that you want to publish.
- Remote Data Topic (optional): Enter the name of a Databricks table to publish this topic's data to. If left empty, data is published to the Default Table defined at the connector level.
- Enable: Select the toggle to enable the topic.
- Click OK to add the topic.
- From the connector tile, verify that the connector shows a CONNECTED status and the topic shows an Enabled status.
Step 4: Verify Data in Databricks
To confirm that data is flowing from Litmus Edge into Databricks:
- In your Databricks workspace, navigate to the target catalog, schema, and table.
- Run a query against the table to view incoming rows:
SELECT * FROM <catalog>.<schema>.<table>
ORDER BY timestamp DESC
LIMIT 100;- Confirm that rows appear with values from your Litmus Edge tags.
Tip: You can also verify ingestion activity from the Databricks SQL Editor or by monitoring the table's data freshness in Unity Catalog.