Microsoft Azure Storage

This page shows how to set up and use the Microsoft Azure Storage destination. The Microsoft Azure Storage destination loads data to a cloud based Azure Storage.

Requirements

The Azure Storage (Blob / Data Lake) destination supports the following Azure storage account types:

• General-purpose V2 (including Azure Data Lake Storage Gen2)
• General-purpose V1
• BlockBlobStorage
• BlobStorage

To use the Azure Storage (Blob / Data Lake) destination you need one of the above Azure storage accounts. For more information, see Microsoft Documentation: Azure storage account overview.

Create a new Microsoft Azure Storage Destination

Follow the steps below to add a new Microsoft Azure Storage destination to Xtract Universal:

1. In the main window of the Designer, navigate to Server > Manage Destinations. The window “Manage Destinations” opens.
   
2. Click [Add] to create a new destination. The window "Destination Details" opens.
   
3. Enter a Name for the destination.
4. Select the destination type Microsoft Azure Storage from the drop-down menu. A list of connection details opens.
5. Fill out the destination details to connect to the destination.
6. Click [OK] to confirm your input.

The destination can now be assigned to extractions.

Destination Details

The destination details define the connection to the destination.

Connection Type

The subsection Connection Type offers the following methods for authenticating and authorizing access to Azure storage:

• Access Key
• Entra ID
• Shared Access Signature (SAS)

For information on advantages and disadvantages of the different authentication methods, see Microsoft Documentation: Choosing the right authentication method.

Access Key Parameters / SAS Parameters

The input fields in the subsection Access key parameters / SAS parameters vary depending on the selected connection type.

Container

This subsection is activated after a connection to the storage account was successfully established.

• When using Access Key authentication, select a Blob container from the drop-down list.
• When using Entry ID authentication, enter the name of the Blob container manually.
• When using Shared Access Signature (Account) authentication, select a Blob container from the drop-down list.

Test connection

Click [Test Connection] to check if the storage container can be accessed.
If the connection is successful, a "Connection to container <name of container> successful" info window opens.

The Azure Storage (Blob / Data Lake) destination can now be used.

Misc

Folder path

Creates a folder structure within the container. Script expressions are supported, see Use Script Expressions as Dynamic Folder Paths.

• Create a single folder by entering a folder name without slashes: [folder]
  
• Create subfolders using the following syntax: [folder]/[subfolder_1]/[subfolder_2]/[..]

File Format

Select the required file format. You can choose between CSV, JSON and Parquet.

CVS Settings

The settings for file type CSV correspond to the settings of the Flat File CSV destination:

• CSV Settings
• Convert / Encoding

JSON Settings

To use the JSON file format, no further settings are necessary.

Parquet Settings

The settings for file type Parquet correspond to the settings of the Flat File Parquet destination:

• Compatibility Mode

Connection Retry and Rollback

Connection retry is a built-in function of the Microsoft Azure Storage destination. The retry function is activated by default.

Connection retry is a functionality that prevents extractions from failing in case of transient connection interruptions to Microsoft Azure Storage. Xtract Universal follows an exponential retry strategy. The selected exponential strategy results in seven retry attempts and an overall timespan of 140 seconds. If a connection is not established during the timespan of 140 seconds, the extraction fails.

The retry function is applied after receiving one of the following HTTP errors or exceptions:

• 503 Service Unavailable
• 504 Gateway Timeout
• WebExceptionStatus.ConnectionClosed
• WebExceptionStatus.ConnectFailure
• WebExceptionStatus.Timeout
• WebExceptionStatus.RequestCanceled
• WebExceptionStatus.SendFailure
• WebExceptionStatus.NameResolutionFailure

Rollback covers scenarios where extractions do not fail due to connection failures to Azure but due to an error when connecting to SAP. In those cases Xtract Universal tries to remove any files from Azure storage that were created in the course of the extraction.

Assign the Microsoft Azure Storage Destination to an Extraction

Extractions write data to their assigned destination. Follow the steps below to assign a destination to an extraction:

1. In the main window of the Designer, select an extraction.
2. Click [ Destination]. The window “Destination Settings” opens.
   
3. In the “Destination Settings” window, select a destination from the dropdown list.
   
4. Optional: edit the destination settings.
5. Click [OK] to confirm your input.

When running the extraction, the extracted SAP data is now written to the destination.

Destination Settings

The destination settings only affect the extraction that the destination is assigned to. To open the destination settings, select an extraction in the main window of the Designer and click [ Destination]. The window "Destination Settings" opens.

File Name

Determines the name of the target table. The following options are available:

Append timestamp

Add the timestamp in the UTC format (_YYYY_MM_DD_hh_mm_ss_fff) to the file name of the extraction

Use Script Expressions as Dynamic File Names

Script expressions can be used to generate a dynamic file name. This allows generating file names that are composed of an extraction's properties, e.g. extraction name, SAP source object. This scenario supports script expressions based on .NET and the following XU-specific custom script expressions:

#{Source.Name}#

#{Extraction.ExtractionName}#

#{Extraction.FullyQualifiedExtractionName}#

#{Extraction.Type}#

#{Extraction.SapObjectName}#

#{Extraction.Timestamp}#

#{Extraction.SapObjectName.TrimStart("/".ToCharArray())}#

#{Extraction.SapObjectName.Replace('/', '_')}#

#{Extraction.Fields["FISCPER"].RangeSelections[0].LowerValue}#

#{Extraction.Fields["FISCPER"].RangeSelections[0].UpperValue}#

#{Extraction.Context}#

#{Extraction.Fields["[NameSelectionFiels]"].Selections[0].Value}#

#{Odp.UpdateMode}#

#{TableExtraction.WhereClause}#

Column Name Style

Defines the style of the column name. Following options are available:

Convert dates

Converts the character-type SAP date (YYYYMMDD, e.g., 19900101) to a special date format (YYYY-MM-DD, e.g., 1990-01-01). Target data uses a real date data-type and not the string data-type to store dates.

Year 0

Converts the SAP date 00000000 to the entered value.

Year 9999

Converts the SAP date 9999XXXX to the entered value.

Invalid values

If an SAP date cannot be converted to a valid date format, the invalid date is converted to the entered value. NULL is supported as a value. When converting the SAP date the two special cases 00000000 and 9999XXXX are checked first.

Blob Type

Append Blob

Creates an Append Blob.

Block Blob

Creates a Block Blob.

Folder

Writes the extracted data to a specific folder structure within the container. If the specified folder does not exist, it is created.

• Create a single folder by entering a folder name without slashes: [folder]
  
• Create subfolders using the following syntax: [folder]/[subfolder_1]/[subfolder_2]/[..]

Use Script Expressions as Dynamic Folder Paths

Script expressions can be used to generate a dynamic folder path. This allows generating folder paths that are composed of an extraction's properties, e.g., extraction name, SAP source object. The described scenario supports script expressions based on .NET and the following XU-specific custom script expressions:

#{Source.Name}#

#{Extraction.ExtractionName}#

#{Extraction.FullyQualifiedExtractionName}#

#{Extraction.Type}#

#{Extraction.SapObjectName}#

#{Extraction.Timestamp}#

#{Extraction.SapObjectName.TrimStart("/".ToCharArray())}#

#{Extraction.SapObjectName.Replace('/', '_')}#

#{Extraction.Context}#

#{Extraction.Fields["[NameSelectionFields]"].Selections[0].Value}#

#{Odp.UpdateMode}#

#{TableExtraction.WhereClause}#

#{Extraction.Fields["[0D_NW_CODE]"].Selections[0].Value}#

#{Extraction.Fields["[0D_NW_CHANN]"].RangeSelections[0].LowerValue}#

#{Extraction.Fields["[0D_NW_CHANN]"].RangeSelections[0].UpperValue}#

#{Extraction.Fields["0D_NW_CODE"].Selections[0].Value}#

#{Extraction.Fields["0D_NW_CHANN"].RangeSelections[0].LowerValue}#

#{Extraction.Fields["0D_NW_CHANN"].RangeSelections[0].UpperValue}#

Compression

Compression is only available for the csv file format, see Destination Details: File Format.

GZip

The data is transferred compressed and stored as a gz file.

File Splitting

Writes extraction data of a single extraction to multiple files. Each filename is appended by _part[nnn].

Max. file size

The value set in Max. file size determines the maximum size of each file.

Empty Files

When this option is active, empty result sets create an empty file in the target environment. Deactivate this option to not create empty files.

Tutorial

The following YouTube tutorial shows how to set up Xtract Universal with the Azure Storage destination:

Related Links

• Knowledge Base Article: Authentication via Microsoft Entra ID for Azure Storage
• Integration via Azure Data Factory