Databricks Zerobus Ingest Integration Guide

note this feature is available starting with litmus edge 4 1 x use this guide to set up an integration between litmus edge and databricks zerobus ingest https //docs databricks com/aws/en/ingestion/zerobus overview once the integration is set up, you can use it to publish data from litmus edge directly into unity catalog delta tables using the zerobus ingestion pipeline note this connector publishes data from litmus edge to databricks only inbound subscriptions are not supported before you begin before configuring the connector in litmus edge, complete the following setup steps in databricks see the following databricks resources to learn more about zerobus ingest and configuring your workspace databricks zerobus ingest overview https //docs databricks com/aws/en/ingestion/zerobus overview create an external location https //docs databricks com/aws/en/connect/unity catalog/external locations html manage service principals https //docs databricks com/aws/en/admin/users groups/service principals html unity catalog privileges and securable objects https //docs databricks com/aws/en/data governance/unity catalog/manage privileges/privileges html step 1 create a cloud storage bucket create a cloud storage bucket or container appropriate for your databricks deployment for example, use an aws s3 bucket, azure blob storage container, or google cloud storage bucket this storage will serve as the external location for your databricks catalog refer to your cloud provider's documentation for creating a bucket or container and note the bucket or container uri for use in the next step step 2 create an external location in databricks in your databricks workspace, create an external location and link it to the cloud storage bucket created in step 1 see create an external location https //docs databricks com/aws/en/connect/unity catalog/external locations html for detailed instructions step 3 create a service principal and oauth secret create a service principal in your databricks workspace, then generate an oauth secret for it litmus edge uses these credentials to authenticate with the databricks zerobus ingest endpoint after creation, record the following values you will need them when configuring the connector in litmus edge client id the unique identifier for the service principal, in uuid format (for example xxxxxxxx xxxx xxxx xxxx xxxxxxxxxxxx ) oauth secret the secret generated for the service principal important store the oauth secret securely it is displayed only once at creation time see manage service principals https //docs databricks com/aws/en/admin/users groups/service principals html for detailed instructions step 4 create a catalog in your databricks workspace, create a new catalog backed by the external location created in step 2 important do not use a catalog whose storage location is set to default storage the zerobus ingest connector requires a catalog backed by an explicit external location see create catalog https //docs databricks com/aws/en/catalogs/create catalog for detailed instructions step 5 create a schema in the catalog, create a new schema or use the automatically created default schema see create schemas https //docs databricks com/aws/en/schemas/create schema for detailed instructions step 6 create a table create a table in the schema with columns that match the data fields litmus edge will publish the following example creates a table compatible with the litmus edge devicehub tag payload create table if not exists \<catalog> \<schema> \<table> ( success boolean, datatype string, timestamp bigint, registerid string, value double, deviceid string, tagname string, devicename string, description string, metadata variant ); replace \<catalog> , \<schema> , and \<table> with the names you chose in steps 4 and 5 for more information on table creation syntax and identifier naming restrictions, see create table statement https //docs databricks com/aws/en/sql/language manual/sql ref syntax ddl create table using html databricks identifiers https //docs databricks com/aws/en/sql/language manual/sql ref identifiers html note this guide uses devicehub tag payload as an example for the databricks table schema and outbound topic configuration litmus edge also supports streaming custom json structures produced by analytics or digital twins to databricks the steps are the same, but you will need to define your databricks sql table columns to match your custom payload structure step 7 grant permissions to the service principal grant the service principal created in step 3 the necessary permissions on the catalog and schema run the following sql statements in your databricks workspace, replacing \<catalog> , \<schema> , \<table> , and \<client id> with your values grant use catalog on catalog \<catalog> to `\<client id>`; grant use schema on schema \<catalog> \<schema> to `\<client id>`; grant modify, select on table \<catalog> \<schema> \<table> to `\<client id>`; see unity catalog privileges and securable objects https //docs databricks com/aws/en/data governance/unity catalog/manage privileges/privileges html for more information set up the outbound connection (publish to databricks) follow the steps below to configure litmus edge to publish data to databricks step 1 add the databricks zerobus ingest connector follow the steps to add a connector and select the databricks zerobus ingest connector provider configure the following parameters name the name of the connector workspace url the url of your databricks workspace (for example https //\<workspace id> cloud databricks com ) zerobus ingest url the zerobus ingestion endpoint for your workspace the format depends on your cloud provider aws \<workspace id> zerobus \<region> cloud databricks com (see zerobus ingest on aws https //docs databricks com/aws/en/ingestion/zerobus ingest ) azure \<workspace id> zerobus \<region> azuredatabricks net (see zerobus ingest on azure https //learn microsoft com/en us/azure/databricks/ingestion/zerobus ingest ) client id the client id of the service principal created in the before you begin section oauth secret the oauth secret generated for the service principal catalog the name of the databricks catalog created in the before you begin section schema the name of the schema within the catalog (for example default ) default table the name of the target table within the schema step 2 enable the connector after adding the connector, click the toggle in the connector tile to enable it if you see a failed status, review the manage connectors docid\ beqq0sszwsv9v9rif3rhk page and the relevant error messages step 3 add an outbound topic after the connector is enabled, add one or more outbound topics to define which litmus edge data is published to databricks to add an outbound topic click the connector tile the connector dashboard appears click the topics tab click the add a new topic icon the data integration dialog box appears configure the following parameters data direction select local to remote outbound local data topic select the devicehub tag or topic from litmus edge that you want to publish remote data topic (optional) enter the name of a databricks table to publish this topic's data to if left empty, data is published to the default table defined at the connector level enable select the toggle to enable the topic click ok to add the topic from the connector tile, verify that the connector shows a connected status and the topic shows an enabled status step 4 verify data in databricks to confirm that data is flowing from litmus edge into databricks in your databricks workspace, navigate to the target catalog, schema, and table run a query against the table to view incoming rows select from \<catalog> \<schema> \<table> order by timestamp desc limit 100; confirm that rows appear with values from your litmus edge tags tip you can also verify ingestion activity from the databricks sql editor or by monitoring the table's data freshness in unity catalog