ADLS Gen2 Writer
Writes to files in an Azure Data Lake Storage Gen2 file system. A common use case is to write data from on-premise sources to an ADLS staging area from which it can be consumed by Azure-based analytics tools.
When you create the Gen2 storage account, set Storage account kind to StorageV2 and enable Hierarchical namespace.
ADLS Gen2 Writer properties
property | type | default value | notes |
---|---|---|---|
Account Name | String | the storage account name | |
Compression Type | String | Set to | |
Directory | String | The full path to the directory in which to write the files. See Setting output names and rollover / upload policies for advanced options. | |
File Name | String | The base name of the files to be written. See Setting output names and rollover / upload policies. | |
File System Name | String | the name of the ADLS Gen2 file system (container) where the files will be written | |
Rollover on DDL | Boolean | True | Has effect only when the input stream is the output stream of a MySQLReader or OracleReader source. With the default value of True, rolls over to a new file when a DDL event is received. Set to False to keep writing to the same file. |
SAS Token | encrypted password | The SAS token for a shared access signature for the storage account. Allowed services must include Blob, allowed resource types must include Object, and allowed permissions must include Write and Create. Remove the If a running Striim Cloud private endpoint is associated with the same Azure service as the SAS token, Striim will use it automatically (see Using Azure private endpoints for more information). | |
Upload Policy | String |
| See Setting output names and rollover / upload policies. Keep these settings low enough that individual uploads do not exceed the underlying Microsoft REST API's limit of 100 MB for a single operation. For best performance, Microsoft recommends uploads between 4 and 16 MB. Setting UploadPolicy to When the app is stopped, any remaining data in the upload buffer is discarded. |
This adapter has a choice of formatters. See Supported writer-formatter combinations for more information.
ADLS Gen2 Writer sample application
CREATE APPLICATION ADLSGen2Test; CREATE SOURCE PosSource USING FileReader ( wildcard: 'PosDataPreview.csv', directory: 'Samples/PosApp/appData', positionByEOF:false ) PARSE USING DSVParser ( header:Yes, trimquote:false ) OUTPUT TO PosSource_Stream; CREATE CQ PosSource_Stream_CQ INSERT INTO PosSource_TransformedStream SELECT TO_STRING(data[1]) AS MerchantId, TO_DATE(data[4]) AS DateTime, TO_DOUBLE(data[7]) AS AuthAmount, TO_STRING(data[9]) AS Zip FROM PosSource_Stream; CREATE TARGET ADLSGen2Target USING ADLSGen2Writer ( accountname:'mystorageaccount', sastoken:'********************************************', filesystemname:'myfilesystem', directory:'mydir', filename:'myfile.json', uploadpolicy: 'interval:15s' ) FORMAT USING JSONFormatter () INPUT FROM PosSource_TransformedStream; END APPLICATION ADLSGen2Test;