Google Cloud Storage

This page shows how to set up and use the Google Cloud Storage destination. The Google Cloud Storage destination loads data to a Google Cloud Storage.

About Google Cloud Storage

Google Cloud Platform (GCP) is a collection of cloud services provided by Google. Google Cloud Platform is available at cloud.google.com.

Google Cloud Storage is one of the Google services used for storing data in the Google infrastructure. For more information see Google Cloud Storage Documentation.

GCP console

The GCP console allows configuring of all resources and services. To get to the overview dashboard, navigate to the Google Cloud Storage page and click [Console] or [Go to console].

To access all settings and services use the navigation menu on the upper left side.

Requirements

• A Google account
• A Google Cloud Platform (GCP) subscription (trial version offered)
• A project ("My First Project" is pre-defined)
• A Google Cloud Storage (GCS) bucket for data extractions

Create a new Google Cloud Storage Destination

Follow the steps below to add a new Google Cloud Storage destination to Xtract Universal:

1. In the main window of the Designer, navigate to Server > Manage Destinations. The window “Manage Destinations” opens.
   
2. Click [Add] to create a new destination. The window "Destination Details" opens.
   
3. Enter a Name for the destination.
4. Select the destination type Google Cloud Storage from the drop-down menu. A list of connection details opens.
5. Fill out the destination details to connect to the destination.
6. Click [OK] to confirm your input.

The destination can now be assigned to extractions.

Destination Details

The destination details define the connection to the destination.

GCS Settings

Connection Type

The subsection Connection Type offers two different methods for authenticating and authorizing access to an Azure Storage account:

The following input fields vary depending on the selected authentication method.

Bucket

When using OAuth 2.0 authentication, the "Bucket" subsection can only be filled after a connection to the storage account has been established.

Project ID

The Project ID can be looked up in the GCP dashboard under Project info.

Bucket name

When using OAuth 2.0 authentication, click [Get buckets] to display available buckets.
A bucket can be created in the navigation menu under Storage > Browser.

Choose a bucket name, location type and storage class or access control.

Under Advanced Settings (optional) you can select the desired encryption method applied to the bucket. Get more details on encryption on the official Google Help Page.

Encryption

Default

Applies the encryption method specified in your GCS bucket.
Google encrypts all data that is stored on the Google servers by default. In addition you can use the Google Cloud Key Management Service (KMS) to create and apply keys to your buckets.
The KMS can be enabled in the GCP console's navigation menu under Security > Cryptographic Keys.

Customer-supplied

If you check the Customer-supplied option, you need to provide a valid AES256 Crypto Key (256 bit in length).
The Crypto key is not stored in the GCP and demands the additional effort to be able to to decrypt your data later.

Crypto key

If Customer Supplied is selected as the encryption method, enter the cryptographic key into.

Misc

Folder path

Creates a folder structure within the bucket. Script expressions are supported, see Use Script Expressions as Dynamic Folder Paths.

• Create a single folder by entering a folder name without slashes: [folder]
  
• Create subfolders using the following syntax: [folder]/[subfolder_1]/[subfolder_2]/[..]

File Format

Select the required file format. You can choose between CSV, JSON and Parquet.

CVS Settings

The settings for file type CSV correspond to the settings of the Flat File CSV destination:

• CSV Settings
• Convert / Encoding

JSON Settings

To use the JSON file format, no further settings are necessary.

Parquet Settings

The settings for file type Parquet correspond to the settings of the Flat File Parquet destination:

• Compatibility Mode

Connection Retry

Connection retry is a built-in function of the Google Cloud Storage destination. The retry function is activated by default.

Connection retry is a functionality that prevents extractions from failing in case of transient connection interruptions to Google Cloud Storage. Xtract Universal follows an exponential retry strategy. The selected exponential strategy results in eight retry attempts and an overall timespan of 140 seconds. If a connection is not established during the timespan of 140 seconds, the extraction fails.

The retry function is applied after receiving one of the following HTTP errors or exceptions:

• 503 Service Unavailable
• 504 Gateway Timeout
• WebExceptionStatus.ConnectionClosed
• WebExceptionStatus.ConnectFailure
• WebExceptionStatus.Timeout
• WebExceptionStatus.RequestCanceled
• WebExceptionStatus.SendFailure
• WebExceptionStatus.NameResolutionFailure

Assign the Google Cloud Storage Destination to an Extraction

Extractions write data to their assigned destination. Follow the steps below to assign a destination to an extraction:

1. In the main window of the Designer, select an extraction.
2. Click [ Destination]. The window “Destination Settings” opens.
   
3. In the “Destination Settings” window, select a destination from the dropdown list.
   
4. Optional: edit the destination settings.
5. Click [OK] to confirm your input.

When running the extraction, the extracted SAP data is now written to the destination.

Destination Settings

The destination settings only affect the extraction that the destination is assigned to. To open the destination settings, select an extraction in the main window of the Designer and click [ Destination]. The window "Destination Settings" opens.

File Name

Determines the name of the target table. The following options are available:

Append timestamp

Add the timestamp in the UTC format (_YYYY_MM_DD_hh_mm_ss_fff) to the file name of the extraction

Use Script Expressions as Dynamic File Names

Script expressions can be used to generate a dynamic file name. This allows generating file names that are composed of an extraction's properties, e.g. extraction name, SAP source object. This scenario supports script expressions based on .NET and the following XU-specific custom script expressions:

#{Source.Name}#

#{Extraction.ExtractionName}#

#{Extraction.FullyQualifiedExtractionName}#

#{Extraction.Type}#

#{Extraction.SapObjectName}#

#{Extraction.Timestamp}#

#{Extraction.SapObjectName.TrimStart("/".ToCharArray())}#

#{Extraction.SapObjectName.Replace('/', '_')}#

#{Extraction.Fields["FISCPER"].RangeSelections[0].LowerValue}#

#{Extraction.Fields["FISCPER"].RangeSelections[0].UpperValue}#

#{Extraction.Context}#

#{Extraction.Fields["[NameSelectionFiels]"].Selections[0].Value}#

#{Odp.UpdateMode}#

#{TableExtraction.WhereClause}#

Column Name Style

Defines the style of the column name. Following options are available:

Convert dates

Converts the character-type SAP date (YYYYMMDD, e.g., 19900101) to a special date format (YYYY-MM-DD, e.g., 1990-01-01). Target data uses a real date data-type and not the string data-type to store dates.

Year 0

Converts the SAP date 00000000 to the entered value.

Year 9999

Converts the SAP date 9999XXXX to the entered value.

Invalid values

If an SAP date cannot be converted to a valid date format, the invalid date is converted to the entered value. NULL is supported as a value. When converting the SAP date the two special cases 00000000 and 9999XXXX are checked first.

Folder

Writes the extracted data to a specific folder structure within the bucket. If the specified folder does not exist, it is created.

• Create a single folder by entering a folder name without slashes: [folder]
  
• Create subfolders using the following syntax: [folder]/[subfolder_1]/[subfolder_2]/[..]

Use Script Expressions as Dynamic Folder Paths

Script expressions can be used to generate a dynamic folder path. This allows generating folder paths that are composed of an extraction's properties, e.g., extraction name, SAP source object. The described scenario supports script expressions based on .NET and the following XU-specific custom script expressions:

#{Source.Name}#

#{Extraction.ExtractionName}#

#{Extraction.FullyQualifiedExtractionName}#

#{Extraction.Type}#

#{Extraction.SapObjectName}#

#{Extraction.Timestamp}#

#{Extraction.SapObjectName.TrimStart("/".ToCharArray())}#

#{Extraction.SapObjectName.Replace('/', '_')}#

#{Extraction.Context}#

#{Extraction.Fields["[NameSelectionFields]"].Selections[0].Value}#

#{Odp.UpdateMode}#

#{TableExtraction.WhereClause}#

#{Extraction.Fields["[0D_NW_CODE]"].Selections[0].Value}#

#{Extraction.Fields["[0D_NW_CHANN]"].RangeSelections[0].LowerValue}#

#{Extraction.Fields["[0D_NW_CHANN]"].RangeSelections[0].UpperValue}#

#{Extraction.Fields["0D_NW_CODE"].Selections[0].Value}#

#{Extraction.Fields["0D_NW_CHANN"].RangeSelections[0].LowerValue}#

#{Extraction.Fields["0D_NW_CHANN"].RangeSelections[0].UpperValue}#

Compression

Compression is only available for the csv file format, see Destination Details: File Format.

File Splitting

Writes extraction data of a single extraction to multiple files. Each filename is appended by _part[nnn].

Max. file size

The value set in Max. file size determines the maximum size of each file.

Empty Files

When this option is active, empty result sets create an empty file in the target environment. Deactivate this option to not create empty files.