Apache Iceberg

This page shows how to use the Apache Iceberg destination to write data into an S3 storage using the open table format (OTF) Iceberg Table Format v2. The Apache Iceberg destination currently only supports the Amazon S3 target environment.

The Apache Iceberg destination:

• writes SAP data as Parquet files into an Amazon S3 storage.
• automatically creates and maintains Apache Iceberg metadata (manifests, snapshots)
• registers tables in REST-based Iceberg catalogs to enable querying data by engines like Spark, Trino, Athena, and Snowflake (read-only).

The setup in Xtract Universal.iQ consists of 3 main steps:

Connect Apache Iceberg

To create a connection for Apache Iceberg:

1. Open the Connections menu.
2. Click [Add Connection].
3. Enter a name in the Connection Name field.
4. Select Apache Iceberg from the Connection Type dropdown. The corresponding settings open.
5. Fill out the connection settings.
6. Click [Test Connection] to validate the connection parameters. A status message opens in the bottom right corner of the window.
7. If the connection is successful, click [Save].

The connection is now listed in the Connections menu and can be assigned to services.

Next, assign the destination to a service.

Assign Apache Iceberg to a Service

To assign the Apache Iceberg destination to a service:

1. Open the Services menu.
2. Select one or more services.
3. Click [ Edit].
4. Select an existing Apache Iceberg connection from the Destination dropdown.
5. Click [Save].

When running the service, the data is written to the selected object storage in an open table format.

Next, check if the service-specific destination settings match your needs.

Connection Settings

Connection settings define how Xtract Universal.iQ connects to Apache Iceberg. To open the connection settings:

1. Open the Connections menu.
2. Select a connection of type Apache Iceberg.
3. Click [ Edit].

The connection settings of the Apache Iceberg connection open and can be modified.

Catalog Options

The Catalog section contains settings for the Iceberg catalog that manages the metadata of the Iceberg tables.

Namespace

Enter the name of a namespace for the Iceberg tables. If the namespace does not exist, a new namespace is created.

Catalog Type

The catalog type defines which metadata service is used to manage the Apache Iceberg tables. Xtract Universal.iQ supports the following catalog types:

• S3Table Bucket: uses Amazon S3 Tables to manage Iceberg tables directly inside the S3 storage.
• AWSGLUE: stores Iceberg table metadata in the AWS Glue Data Catalog for compatibility with Athena, EMR, Databricks, and other analytics tools.

Account ID

This option is only available for the S3Table Bucket catalog type. Enter the ID of the AWS account that owns the S3 bucket where the Iceberg metadata is stored. You can find the account ID in the profile section of the AWS console.

Warehouse

Enter a folder path to write data to a specific folder structure within the S3 bucket. If the specified folder does not exist, it is created.

• Create a single folder by entering a folder name without slashes: folder
  
• Create subfolders using the following syntax: folder/subfolder_1/subfolder_2/...

Authentication

The Authentication section contains settings that define how to connect to the Amazon S3 target environment. Apache Iceberg uses AWS access keys for the authentication. The keys are managed by AWS Identity and Access Management (IAM), see AWS Documentation: Manage access keys for IAM users.

Access Key

The access key ID is a 20-character, alphanumeric string, e.g., AKIAIOSFODNN7EXAMPLE. You can find or generate access keys in the AWS Management Console by navigating to:
IAM > Users > [Your User] > Security credentials.

Secret Key

The secret access key is a 40-character string, e.g., wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY. Note that secret keys are shown only once at creation time. Make sure to copy them and store them in a safe location.

Connect

Click [Connect] to validate your connection settings. A status message is displayed at the bottom right corner of the screen.

Bucket

The Bucket section contains settings that define which bucket to connect to.

Region

Select the region of your Amazon S3 bucket.

Bucket

The Bucket dropdown becomes available once a bucket region is selected. Click to load or refresh the list of available buckets in Amazon S3. Select a bucket from the drop-down list. The SAP data is written into the selected bucket.

Destination Settings

Destination settings define how and where a service saves its output data, such as the target folder and filename.

To open the destination settings:

1. Open the Services menu.
2. Select a service that uses the Apache Iceberg connection.
3. Click [ Edit]. The service settings open.
4. Open the tab Destination Settings.

The destination settings for Apache Iceberg are open and can be modified.

File Name

Determines the name of the target table. The following options are available:

Append timestamp

Adds a timestamp in the format _YYYY_MM_DD to the filename of the service.

Column Name Format

Defines the format of the column name. The following options are available:

Iceberg Settings

Include Previous Snapshots

When enabled, new data is written on top of the existing Iceberg table state. This preserves data from previous snapshots, e.g., rollback scenarios. Use this option for incremental loads.

Date Conversion

Convert SAP Dates

Converts the SAP date format (YYYYMMDD) to a regular date format (YYYY-MM-DD), e.g., 19900101 becomes 1990-01-01. Target data uses a real date data-type and not the string data-type to store dates.

Replace SAP Dates 0000XXXX with

Converts the SAP date 00000000 to the entered value.

Replace SAP Dates 9999XXXX with

Converts the SAP date 9999XXXX to the entered value.

Replace Invalid SAP Dates with

If an SAP date cannot be converted to a valid date format, the invalid date is converted to the entered value. NULL is supported as a value. When converting the SAP date the two special cases 00000000 and 9999XXXX are checked first.