DB - Databricks SQL (Gen 1)
Overview
The DB - Databricks SQL (Gen 1) integration enables an Edge System to connect with Databricks.
Supported Communication
- Outbound (Write from Edge to Databricks) - Yes
- Inbound (Read from Databricks to Edge) - No
Configurations
Variable | Detail | Required |
|---|---|---|
Name | Enter a name for the connector | Yes |
Connection String | Connection string for the data source name. For personal token authentication, use the following format: token:personal-access-token@server-hostname:port-number/http-path?param1=value1¶m2=value2 For M2M OAuth authentication, use the following format: server-hostname:port-number/http-path?authType=OAuthM2M&clientID=client-id&clientSecret=client-secret | Yes |
Table | Table Name | Yes |
Mapping | Data from a predefined table could be mapped into a custom one. Example: {"from": "to"} where "from" is a column from predefined table, and "to" is a column from destination table. See Work with Tables in SQL Connectors (Create Table and Show Mapping) to learn more. | No |
Create Table | If selected and table doesn't exist, the table is created with the necessary format. See Work with Tables in SQL Connectors (Create Table and Show Mapping) to learn more. | No |
Bulk insert count | To enable this option, enter the number of messages to group together and send as one bulk insert statement. Enabling this option can improve how quickly data is processed when dealing with high volumes of tags. | No |
Commit Timeout | Timeout (in milliseconds) for committing bulk inserts. | No |
Throttling limit | Set a limit on the number of outbound messages per second. If sending data to a server that has restrictions on the rate of incoming messages, this ensures that no messages are lost. | No |
Persistent storage | If enabled, messages in the outbound queue are stored on disk. If there are interruptions in connection or power loss, messages will be retained and eventually sent when the connection is restored. | No |
Queue mode | Select the sequence of data transfer: FIFO or LIFO. Selecting FIFO (First In First Out) will send the earliest message first. Selecting LIFO (Last In First Out) will send the latest message first. | No |