Create a Cloud Storage Sync Job

4 min

you can create cloud storage sync jobs by navigating to integration > object before you begin before you can start leveraging cloud storage sync, you will need to s et up the source directory see docid\ eet4fyvq5gyaupphkuaf7 to learn more create cloud storage sync job to create a cloud storage sync job navigate to integration > object click add sync job the add cloud sync job dialog box displays enter a name for the job select a provider for the job configure the parameters for the provider aws s3 region enter the region name see https //docs aws amazon com/general/latest/gr/s3 html to learn more access key id enter the appropriate access key id see https //docs aws amazon com/iam/latest/userguide/security creds html to learn more secret access key enter the appropriate secret access key see https //docs aws amazon com/iam/latest/userguide/security creds html to learn more source enter the file path to the local directory for example /var/lib/customer/ftp data/s3 sync make sure you have started the litmus edge ftp server and created the necessary folders for the file path see docid\ eet4fyvq5gyaupphkuaf7 to learn more destination enter the remote s3 bucket name transfer mode select a transfer mode copy or move run every configure the time interval for the sync job in minutes, hours, or seconds azure blob storage access key enter the access key see the following resources to learn more https //learn microsoft com/en us/azure/storage/blobs/security recommendations https //learn microsoft com/en us/azure/key vault/general/overview access name enter the storage account name source enter the file path to the local directory for example /var/lib/customer/ftp data/azure sync make sure you have started the litmus edge ftp server and created the necessary folders for the file path see docid\ eet4fyvq5gyaupphkuaf7 to learn more destination enter the azure blob storage container name transfer mode select a transfer mode copy or move run every configure the time interval for the sync job in minutes, hours, or seconds google cloud storage json key file pase or upload the json key file source enter the file path to the local directory for example /var/lib/customer/ftp data/gcp sync make sure you have started the litmus edge ftp server and created the necessary folders for the file path see docid\ eet4fyvq5gyaupphkuaf7 to learn more destination enter the remote gcp bucket name transfer mode select a transfer mode copy or move run every configure the time interval for the sync job in minutes, hours, or seconds databricks unity catalog volume workspace url enter the databricks workspace url see https //docs databricks com/en/workspace/workspace details html to learn more authentication option personal access token if you select this, then access token you can paste the access token here see https //docs databricks com/en/dev tools/auth/pat html to learn more source enter the file path to the local directory for example /var/lib/customer/ftp data/databricks sync make sure you have started the litmus edge ftp server and created the necessary folders for the file path see docid\ eet4fyvq5gyaupphkuaf7 to learn more destination enter the file path to databricks directory transfer mode select a transfer mode copy or move run every configure the time interval for the sync job in minutes, hours, or seconds see docid\ mvwjuqs9bl4nlstpml9w7 guide to configure the databricks unity catalog volume on the litmus edge webui to send files directly to the databricks unity catalog when done configuring the provider, click save once the job is created, you can enable or disable it to manage the job, you must disable the job first