Amazon S3

This page shows how to set up and use the Amazon S3 destination. The Amazon S3 destination loads data to the Amazon S3 cloud storage.
For more information on Amazon S3, see AWS Documentation: Getting Started with Amazon S3.

Requirements

• An Amazon Web Services (AWS) Account.
• Access Keys (consisting of "access key ID" and "secret access key") of your AWS user or an IAM role attached to the EC2 instance that runs Xtract Universal, see AWS Documentation: Using an IAM role to grant permissions to applications running on Amazon EC2 instances. For more information on the IAM role, see AWS Documentation: Security best practices in IAM.
• An S3 bucket, in which you can upload data.
• Sufficient permissions for list, read and write activities on S3. You must grant these rights in the user policy, but you can limit them to certain buckets. In the following example, the set permissions have been tested in a test environment:
  

Extraction Failed

In case the extraction fails, the already uploaded parts are deleted from S3. In case of an "uncontrolled" extraction failure, for example due to network issues, Xtract Universal will not be able to delete those uploaded parts from S3.

Therefore it is recommended to change the settings on S3 in a way that will trigger the automatic deletion of unused multiparts, e.g. after a day. You can find this setting by selecting a bucket and opening the "Management" tab. Select "Lifecycle" and "Add lifecycle rule" and create a rule for deleting unused multiparts.

Create a new Amazon S3 Destination

Follow the steps below to add a new Amazon S3 destination to Xtract Universal:

1. In the main window of the Designer, navigate to Server > Manage Destinations. The window “Manage Destinations” opens.
   
2. Click [Add] to create a new destination. The window "Destination Details" opens.
   
3. Enter a Name for the destination.
4. Select the destination type Amazon S3 from the drop-down menu. A list of connection details opens.
5. Fill out the destination details to connect to the destination.
6. Click [OK] to confirm your input.

The destination can now be assigned to extractions.

Destination Details

The destination details define the connection to the destination.

API Endpoints

The endpoint defines which IP address is used to connect to an Amazon S3 instance.

Use AWS S3 Endpoint

When this option is active, Xtract Universal connects to Amazon S3 using a URL that consists of the bucket name and the Amazon S3 default URL: https://[BucketName].s3.amazonaws.com/

Use VPC Endpoint

When this option is active, Xtract Universal connects to Amazon S3 using a Virtual Private Cloud (VPC) endpoint. This option requires Xtract Universal to run on an AWS VPC instance.

In the field VPC endpoint, enter the VPC endpoint in the format http://bucket.[endpoint]. Example:
http://bucket.vpce-0123456789abcdefg-hijklmno.s3.us-east-1.vpce.amazonaws.com.

Authenthication

The authentication method defines how Xtract Universal authenticates itself against Amazon AWS.

Use IAM role

When this option is active, the credentials and permissions of the IAM role assigned to the EC2 instance, on which Xtract Universal is running are used for authentication. For more information on the IAM role, see AWS Documentation: Security best practices in IAM.

Use user credentials

Preferable authentication method towards Amazon AWS. Determine the values for the Access Keys (access key ID and secret key) via AWS Identity and Access Management (IAM).

Connect

After entering Access key ID and Secret key, click [Connect] to connect to AWS. After successfully connecting, select bucket name and region.

Bucket

Bucket name and Region

Select a bucket and a region of the bucket's location. The SAP data is extracted into the selected bucket.

Test Connection

Validates the right combination of bucket and region. Insures bucket's accessibility from Xtract Universal using the entered access keys.

Server-side encryption

Choose how to encrypt data after uploading them to S3.

None

Server-sided encryption of data not active. For more information, see AWS Documentation: Protecting data with server-side encryption.

SSE-S3

Encrypts data using the by default available S3 user account encryption key, see S3 Managed Encryption Keys.

SSE-KMS / Key ID

Encryption using a custom encryption key created on AWS, see AWS Key Management Services. The key can be created on the AWS website

Misc

All settings in Misc are optional.

Folder path

Creates a folder structure within the bucket. Script expressions are supported, see Use Script Expressions as Dynamic Folder Paths.

• Create a single folder by entering a folder name without slashes: [folder]
  
• Create subfolders using the following syntax: [folder]/[subfolder_1]/[subfolder_2]/[..]

File Owner

If you upload files as an AWS user of an Account A to an Account B, you can select the option "Bucket Owner". Without a declared owner, uploaded files cannot be opened directly.

File Format

Select the required file format. You can choose between CSV, JSON and Parquet.

CVS Settings

The settings for file type CSV correspond to the settings of the Flat File CSV destination:

• CSV Settings
• Convert / Encoding

JSON Settings

To use the JSON file format, no further settings are necessary.

Parquet Settings

The settings for file type Parquet correspond to the settings of the Flat File Parquet destination:

• Compatibility Mode

Connection Retry

Connection retry is a built-in function of the Amazon S3 destination. The retry function is activated by default.

Connection retry is a functionality that prevents extractions from failing in case of transient connection interruptions to Amazon S3. Xtract Universal follows an exponential retry strategy. The selected exponential strategy results in seven retry attempts and an overall timespan of 140 seconds. If a connection is not established during the timespan of 140 seconds, the extraction fails.

The retry function is applied after receiving one of the following HTTP errors or exceptions:

• 503 Service Unavailable
• 504 Gateway Timeout
• WebExceptionStatus.ConnectionClosed
• WebExceptionStatus.ConnectFailure
• WebExceptionStatus.Timeout
• WebExceptionStatus.RequestCanceled
• WebExceptionStatus.SendFailure
• WebExceptionStatus.NameResolutionFailure

For more general information about retry strategies in an AWS S3 environment go to the official AWS Help.

Assign the Amazon S3 Destination to an Extraction

Extractions write data to their assigned destination. Follow the steps below to assign a destination to an extraction:

1. In the main window of the Designer, select an extraction.
2. Click [ Destination]. The window “Destination Settings” opens.
   
3. In the “Destination Settings” window, select a destination from the dropdown list.
   
4. Optional: edit the destination settings.
5. Click [OK] to confirm your input.

When running the extraction, the extracted SAP data is now written to the destination.

Destination Settings

The destination settings only affect the extraction that the destination is assigned to. To open the destination settings, select an extraction in the main window of the Designer and click [ Destination]. The window "Destination Settings" opens.

File Name

Determines the name of the target table. The following options are available:

Append timestamp

Add the timestamp in the UTC format (_YYYY_MM_DD_hh_mm_ss_fff) to the file name of the extraction

Use Script Expressions as Dynamic File Names

Script expressions can be used to generate a dynamic file name. This allows generating file names that are composed of an extraction's properties, e.g. extraction name, SAP source object. This scenario supports script expressions based on .NET and the following XU-specific custom script expressions:

#{Source.Name}#

#{Extraction.ExtractionName}#

#{Extraction.Type}#

#{Extraction.SapObjectName}#

#{Extraction.Timestamp}#

#{Extraction.SapObjectName.TrimStart("/".ToCharArray())}#

#{Extraction.SapObjectName.Replace('/', '_')}#

#{Extraction.Fields["FISCPER"].RangeSelections[0].LowerValue}#

#{Extraction.Fields["FISCPER"].RangeSelections[0].UpperValue}#

#{Extraction.Context}#

#{Extraction.Fields["[NameSelectionFiels]"].Selections[0].Value}#

#{Odp.UpdateMode}#

#{TableExtraction.WhereClause}#

Column Name Style

Defines the style of the column name. Following options are available:

Convert dates

Converts the character-type SAP date (YYYYMMDD, e.g., 19900101) to a special date format (YYYY-MM-DD, e.g., 1990-01-01). Target data uses a real date data-type and not the string data-type to store dates.

Year 0

Converts the SAP date 00000000 to the entered value.

Year 9999

Converts the SAP date 9999XXXX to the entered value.

Invalid values

If an SAP date cannot be converted to a valid date format, the invalid date is converted to the entered value. NULL is supported as a value. When converting the SAP date the two special cases 00000000 and 9999XXXX are checked first.

Folder

Writes the extracted data to a specific folder structure within the bucket. If the specified folder does not exist, it is created.

• Create a single folder by entering a folder name without slashes: [folder]
  
• Create subfolders using the following syntax: [folder]/[subfolder_1]/[subfolder_2]/[..]

Use Script Expressions as Dynamic Folder Paths

Script expressions can be used to generate a dynamic folder path. This allows generating folder paths that are composed of an extraction's properties, e.g., extraction name, SAP source object. The described scenario supports script expressions based on .NET and the following XU-specific custom script expressions:

#{Source.Name}#

#{Extraction.ExtractionName}#

#{Extraction.Type}#

#{Extraction.SapObjectName}#

#{Extraction.Timestamp}#

#{Extraction.SapObjectName.TrimStart("/".ToCharArray())}#

#{Extraction.SapObjectName.Replace('/', '_')}#

#{Extraction.Context}#

#{Extraction.Fields["[NameSelectionFields]"].Selections[0].Value}#

#{Odp.UpdateMode}#

#{TableExtraction.WhereClause}#

#{Extraction.Fields["[0D_NW_CODE]"].Selections[0].Value}#

#{Extraction.Fields["[0D_NW_CHANN]"].RangeSelections[0].LowerValue}#

#{Extraction.Fields["[0D_NW_CHANN]"].RangeSelections[0].UpperValue}#

#{Extraction.Fields["0D_NW_CODE"].Selections[0].Value}#

#{Extraction.Fields["0D_NW_CHANN"].RangeSelections[0].LowerValue}#

#{Extraction.Fields["0D_NW_CHANN"].RangeSelections[0].UpperValue}#

Compression

Compression is only available for the csv file format, see Destination Details: File Format.

GZip

The data is transferred compressed and stored as a gz file.

File Splitting

Writes extraction data of a single extraction to multiple files. Each filename is appended by _part[nnn].

Max. file size

The value set in Max. file size determines the maximum size of each file.

Empty Files

If this option is active, empty result sets create an empty file in the target environment. Deactivate this option to not create empty files.

Related Links

• Knowledge Base Article: Run Xtract Universal in a VM on AWS EC2
• Amazon AWS: Getting Started - Backup & Restore with AWS
• Amazon S3: Getting Started with Amazon S3
• Amazon EC2: Getting Started with Amazon EC2
• Amazon Documentation: Amazon EC2 security groups for Linux instances
• Amazon Documentation: Security best practices in IAM
• Amazon Documentation: Overview of managing access