Skip to content

S3 Compatible Storage

This page shows how to connect to object storages that support the S3 interface, e.g., STACKIT and MinIO. The setup in Xtract Universal.iQ consists of 3 main steps:

  1. Create a connection to an S3 bucket.
  2. Assign the connection to a service.
  3. Define the properties of your output, e.g, filename, column names, etc.

Tip

To connect to an Amazon S3 storage, it is recommended to use the dedicated Amazon S3 destination. The Amazon S3 destination supports more connection and authentication options.

Connect to S3 Compatible Storages

To create a connection to a storage:

  1. Open the Connections menu.
  2. Click [Add Connection].
  3. Enter a name in the Connection Name field.
  4. Select S3 Compatible Storage from the Connection Type dropdown. The corresponding settings open.
  5. Fill out the connection settings.
  6. Click [Test Connection] to validate the connection parameters. A status message opens in the bottom right corner of the window.
  7. If the connection is successful, click [Save].

The connection is now listed in the Connections menu and can be assigned to services.

Next, assign the destination to a service.

Assign S3 Compatible Storage to a Service

To assign the S3 Compatible Storage destination to a service:

  1. Open the Services menu.
  2. Select one or more services.
  3. Click [ Edit].
  4. Select an existing S3 Compatible Storage connection from the Destination dropdown.
  5. Click [Save].

When running the service, the results are written to the S3 bucket specified in the connection.

Next, check if the service-specific destination settings match your needs.

Connection Settings

Connection settings define how Xtract Universal.iQ connects to the storage. To open the connection settings:

  1. Open the Connections menu.
  2. Select a connection of type S3 Compatible Storage.
  3. Click [ Edit].

The connection settings of the S3 Compatible Storage connection open and can be modified.

System

The System section contains settings that define the S3 compatible object storage to connect to.

Endpoint

Enter the endpoint URL of the storage you want to connect to. Examples:

  • AWS: https://s3.us-east-1.amazonaws.com
  • MinIO: http://example.com:9000
  • STACKIT https://object.storage.eu01.onstackit.cloud

Authentication

The Authentication section contains settings that define how to connect to S3 compatible storages. Xtract Universal.iQ uses Access Keys (an access key ID and secret access key) to authenticate with storages.

Access Key

The access key ID is a 20-character, alphanumeric string, e.g., AKIAIOSFODNN7EXAMPLE.

Secret Key

The secret access key is a 40-character string, e.g., wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY. Note that secret keys are shown only once at creation time. Make sure to copy them and store them in a safe location.

Bucket

The Bucket section contains settings that define which bucket to connect to.

Bucket

Enter the name of a bucket inside your storage. The SAP data is written into the bucket.

Test Connection

Click [Text Connection] to validate the connection to your bucket. A status message is displayed at the bottom right corner of the screen.

File Format

The File Format section defines the format of the output file written to the Amazon S3 bucket.

File Type

Select one of the following file formats used for the service output:

  • CSV
  • JSON
  • Parquet

Compatibility mode

For Parquet files, a compatibility mode can be selected to ensure compatibility with different processing engines, such as Spark or Hadoop. The following compatibility modes are available:

  • Pure: standard parquet format

  • Spark: ensures compatibility with Apache Spark by using different data types than the standard parquet format:

    SAP Pure / BigQuery Spark
    INT1 UINT_8 INT16
    TIMS TIME_MILLIS UTF8

  • BigQuery: ensures compatibility with with Google BigQuery by formatting the columns names, see BigQuery Documentation: Column Names.

Note

If an SAP Table contains a 6-character time field of type TIMS (format HHMMSS for hours, minutes and seconds), make sure to use the compatibility mode Spark to ensure compatibility with Microsoft.

CSV Settings

The CSV Settings section is only available for the CSV file type. The settings define the format of the output file written to the bucket.

Column Separator

Defines how two columns in CSV are separated. Typical column separators:

  • , (comma separator)
  • ; (semicolon separator)
  • \t (tab separator)

Row Separator

Defines how two rows in CSV are separated. Typical row separators:

  • \r\n (carriage return + line feed)
  • \n (line feed)
  • \r (carriage return)

Quote Symbol

Defines which character is used to encase field data. A sequence of characters may be used as the quote symbol. Quotation is applied in these scenarios:

  • The column separator is part of the field data.
  • The quote symbol is part of the field data.
  • The row separator is part of the field data.
  • The escape character is part of the field data.

Escape Character

When the escape character is part of the field data, the respective field containing this character is encased by the Quote symbol. The default escape character is the backslash \. The field may remain empty.

Column Names in First Row

Defines if the first row contains the column names. This option is set by default.

Row Separator After Last Row

Defines if the last row contains a row separator. This option is set by default.

Decimal Separator

Defines the decimal separator of decimal number for the output. Dot (.) is the default value.

Date Format

Defines a customized date format (e.g. YYYY-MM-DD or MM/DD/YYYY) for converting valid SAP dates (YYYYMMDD). Default is YYYY-MM-DD.

Time Format

Defines a customized time format (e.g. HH-MM-SS or HH:MM:SS) for converting valid SAP times (HHMMSS). Default is HH:MM:SS.

Misc

Additional connection settings for S3 Compatible Storage. All settings in Misc are optional.

Use Legacy URL Style

When enabled, the connection uses the legacy path-style S3 URL format, e.g., https://storage.example.com/my-bucket/, instead of the modern virtual-hosted-style format, e.g., https://my-bucket.storage.example.com/. Enable this option for storage systems that do not support virtual-hosted-style URLs or require path-style access for compatibility.

Folder Path

Creates a folder structure within the bucket.

  • Create a single folder by entering a folder name without slashes: [folder]
  • Create subfolders using the following syntax: [folder]/[subfolder_1]/[subfolder_2]/[..]

Note

The specified folder path applies to all extractions. To define a folder path for single services, use the Folder option in the Destination Settings.

Destination Settings

Destination settings define how and where a service saves its output data, such as the target folder and filename.

To open the destination settings:

  1. Open the Services menu.
  2. Select a service that uses the S3 Compatible Storage connection.
  3. Click [ Edit]. The service settings open.
  4. Open the tab Destination Settings.

The destination settings for S3 Compatible Storage are open and can be modified.

File Name

Determines the name of the target table. The following options are available:

Option Description
Same as service name Adopt the name of the service.
Custom Define a name of your choice in the Custom File Name input field.

Append timestamp

Adds a timestamp in the format _YYYY_MM_DD to the filename of the service.

Note

If the name of the object does not begin with a letter, it will be prefixed with an ‘x’, e.g. an object by the name _namespace_tabname.csv will be renamed x_namespace_tabname.csv when uploaded to the destination. This is to ensure that all uploaded objects are compatible with Azure Data Factory, Hadoop and Spark, which require object names to begin with a letter or give special meaning to objects whose names start with certain non-alphabetic characters.

Column Name Format

Defines the format of the column name. The following options are available:

Option Description
Code The SAP technical column name is used as column name in the destination e.g., MAKTX.
PrefixedCode The SAP technical column name is prefixed by SAP object name and the tilde character e.g., MAKT~MAKTX
CodeAndText The SAP technical column name and the SAP description separated by an underscore are used as column name in the destination e.g., MAKTX_Material Description (Short Text).
TextAndCode The SAP description and the SAP technical column name description separated by an underscore are used as column name in the destination e.g., Material Description (Short Text)_MAKTX.
Text The SAP description is used as column name in the destination e.g., Material Description (Short Text).

File Settings

File Splitting

Writes extraction data of a single extraction to multiple files. Each filename is appended by _part[nnn].

Max. File Size (in MB)

Defines the maximum size of each splitted file.

Note

The option Max. File Size does not apply to gzip files. The size of a gzipped file cannot be determined in advance.

GZip Compression

When enabled, the data is transferred compressed and stored as a gz file.

Custom Directory

Writes the extracted data to a specific folder structure within the bucket. If the specified folder does not exist, it is created.

  • Create a single folder by entering a folder name without slashes: folder
  • Create subfolders using the following syntax: folder/subfolder_1/subfolder_2/...

Note

The specified folder path only applies to the selected service. To define a folder path for all extractions, use the Folder path option in the Connection Settings.

Script Expressions for Dynamic Folder Paths

Script expressions can be used to generate a dynamic folder path. This allows generating folder paths that are composed of service properties, e.g., service name, SAP source object. The described scenario supports script expressions based on .NET and the following custom script expressions:

Input Description
#{Source.Name}# Name of the SAP source.
#{Extraction.ExtractionName}# Name of the service.
#{Extraction.Type}# Service type (Table, Report, BAPI, etc.).
#{Extraction.SapObjectName}# Name of the SAP object that the service is extracting data from.
#{Extraction.Timestamp}# Timestamp of the service.
#{Extraction.SapObjectName.TrimStart("/".ToCharArray())}# Removes the first slash '/' of an SAP object.
Example: /BIO/TMATERIAL to BIO/TMATERIAL - prevents creating an empty folder in a file path.
#{Extraction.SapObjectName.Replace('/', '_')}# Replaces all slashes '/' of an SAP object.
Example: /BIO/TMATERIAL to _BIO_TMATERIAL - prevents splitting the SAP object name by folders in a file path.
#{TableExtraction.WhereClause}# Only for Table services: returns the WHERE clause of the service.

Create files for empty data sets

When active, empty result sets create an empty file in the target environment. Deactivate this option to not create empty files.

Date Conversion

Convert SAP Dates

Converts the SAP date format (YYYYMMDD) to a regular date format (YYYY-MM-DD), e.g., 19900101 becomes 1990-01-01. Target data uses a real date data-type and not the string data-type to store dates.

Replace SAP Dates 0000XXXX with

Converts the SAP date 00000000 to the entered value.

Replace SAP Dates 9999XXXX with

Converts the SAP date 9999XXXX to the entered value.

Replace Invalid SAP Dates with

If an SAP date cannot be converted to a valid date format, the invalid date is converted to the entered value. NULL is supported as a value. When converting the SAP date the two special cases 00000000 and 9999XXXX are checked first.

Last update: September 17, 2026