Apache Kafka

This page shows connect to Apache Kafka to write SAP data directly to Kafka topics. The setup in Xtract Universal.iQ consists of 3 main steps:

About Apache Kafka

Apache Kafka is a platform for exchanging data between systems in real time. It makes business data available to multiple consumers through a centralized messaging platform. Writing SAP data to Kafka allows applications, reporting tools, and other systems to consume SAP data as soon as it is extracted. Common use cases include real-time analytics, data pipelines, and event-driven integrations.

Kafka stores data as messages within topics. Topics are hosted by Kafka servers called brokers, which store and distribute messages. By distributing data across multiple brokers, Kafka can handle large workloads and provide high availability. To connect to a Kafka environment, clients use one or more bootstrap servers that provide the required information to discover and communicate with all brokers in the Kafka cluster.

The following diagram shows the main Kafka components and how they work together.

flowchart LR

    SAP["SAP Data"]

    subgraph KC["Kafka Cluster"]
        BS["Bootstrap Server<br/>(Initial Connection)"]

        subgraph B[" "]
            BR1["Broker 1"]
            BR2["Broker 2"]
            BR3["Broker 3"]
        end

        T1["Topic: Sales Orders"]
        T2["Topic: Customers"]

        BS --> BR1
        BS --> BR2
        BS --> BR3

        BR1 --> T1
        BR2 --> T1

        BR2 --> T2
        BR3 --> T2
    end

    SAP --> BS

    C1["Reporting Tool"]
    C2["Analytics Platform"]
    C3["Application"]

    T1 --> C1
    T1 --> C2

    T2 --> C2
    T2 --> C3

For more information, see Apache Kafka Documentation.

Connect to Apache Kafka

To create a connection to Apache Kafka:

1. Open the Connections menu.
2. Click [Add Connection].
3. Enter a name in the Connection Name field.
4. Select Apache Kafka from the Connection Type dropdown. The corresponding settings open.
5. Fill out the connection settings.
6. Click [Test Connection] to validate the connection parameters. A status message opens in the bottom right corner of the window.
7. If the connection is successful, click [Save].

The connection is now listed in the Connections menu and can be assigned to services.

Next, assign the destination to a service.

Assign Apache Kafka to a Service

To assign the Apache Kafka destination to a service:

1. Open the Services menu.
2. Select one or more services.
3. Click [ Edit].
4. Select an existing Apache Kafka connection from the Destination dropdown.
5. Click [Save].

When running the service, the data is written to Apache Kafka.

Next, check if the service-specific destination settings match your needs.

Connection Settings

Connection settings define how Xtract Universal.iQ connects to Apache Kafka. To open the connection settings:

1. Open the Connections menu.
2. Select a connection of type Apache Kafka.
3. Click [ Edit].

The connection settings of the Apache Kafka connection open and can be modified.

Server URLs

The Server URLs section contains the connection endpoints for the Kafka broker and the Schema Registry.

Bootstrap server

Enter one or more Kafka bootstrap server addresses using the following format:

<hostname>:9092

A bootstrap server acts as the initial connection point to the Kafka environment. It provides the information required to connect to the available Kafka brokers and publish messages to topics.

Schema Registry Server URL

Optional: Enter the URL of the Schema Registry using the following format:

https://<schema-registry>.<example>.com

The Schema Registry stores and manages message schemas. If schema validation or schema-based message formats are used, Xtract Universal.iQ connects to the Schema Registry to retrieve and register schemas, see Schema Registry Authentication.

Broker Authentication

The Broker Authentication section contains the credentials used to authenticate Xtract Universal.iQ with Kafka brokers.

Access Key ID

Enter the access key ID for the Kafka broker. The access key ID is provided by your Kafka administrator or Kafka service provider. For Confluent Cloud, create an API key and API secret, see Confluent Documentation: Manage API Keys.

Secret Access Key

Enter the secret access key associated with the access key ID. The secret key is provided by your Kafka administrator or Kafka service provider. For Confluent Cloud, create an API key and API secret, see Confluent Documentation: Manage API Keys.

Schema Registry Authentication

The Schema Registry Authentication section contains the credentials used to authenticate Xtract Universal.iQ with the Schema Registry. Schema Registry authentication is only required when a Schema Registry is configured and used to manage message schemas. If no Schema Registry is used, these settings can be left empty.

Access Key ID

Enter the access key ID for the Schema Registry. The access key ID is provided by your Kafka administrator or Schema Registry service provider. For Confluent Cloud, create an API key and API secret, see Confluent Documentation: Manage API Keys.

Secret Access Key

Enter the secret access key associated with the access key ID. The secret access key is provided by your Kafka administrator or Schema Registry service provider. For Confluent Cloud, create an API key and API secret, see Confluent Documentation: Manage API Keys.

Create Records That Do Not Match the Registered Schema

When enabled, the Xtract Universal.iQ produces records even if their schema does not match the schema registered for the subject in the Schema Registry. By default, services stop when a schema mismatch is detected. This behavior prevents invalid records that consumers cannot read.

Enable this option to produce records despite schema validation failures.

Topic

The Topic section contains settings for specifying the Kafka topic that receives extracted SAP data.

Default Topic

Enter the name of a topic. SAP data is written to the default Kafka topic for all services that use this connection. If a different topic is defined in the Destination Settings of the service, it overrides the default topic.

Destination Settings

Destination settings define how and where a service saves its output data, such as the target folder and filename.

To open the destination settings:

1. Open the Services menu.
2. Select a service that uses the Apache Kafka connection.
3. Click [ Edit]. The service settings open.
4. Open the tab Destination Settings.

The destination settings for Apache Kafka are open and can be modified.

File Name

Determines the name of the target table. The following options are available:

Append timestamp

Adds a timestamp in the format _YYYY_MM_DD to the filename of the service.

Kafka Settings

The Kafka Settings section contains Kafka-specific settings for the service.

Topic

Enter the name of a topic. This service writes data to the specified Kafka topic instead of the default topic defined in the connection settings. If the Topic field is empty, the service uses the default topic.

Message Key Column

Select a column whose value is used as the Kafka message key. Kafka uses the message key to decide which partition stores a message. Messages with the same key are written to the same partition. This keeps related data together and maintains the message order within that partition.

Example: If you use the column CustomerID as the message key, all messages for the same customer are written to the same partition.

Schema Name

If Kafka uses the Kafka Schema Registry, enter the name of a schema that is registered in your registry. Xtract Universal.iQ uses this schema when writing data to Kafka.

Schema ID

If Kafka uses the Kafka Schema Registry, enter the ID of the schema. The schema ID identifies the schema version that Kafka consumers use to read the data.

Generate Schema

Click [Generate Schema] to generate a schema based on the values in Schema Name and Schema ID. The generated schema is not automatically registered in the Kafka Schema Registry. To use the generated schema with Kafka, register it manually in your Schema Registry. This allows Kafka administrators to review and manage schemas before they are added to the registry.

Date Conversion

Convert SAP Dates

Converts the SAP date format (YYYYMMDD) to a regular date format (YYYY-MM-DD), e.g., 19900101 becomes 1990-01-01. Target data uses a real date data-type and not the string data-type to store dates.

Replace SAP Dates 0000XXXX with

Converts the SAP date 00000000 to the entered value.

Replace SAP Dates 9999XXXX with

Converts the SAP date 9999XXXX to the entered value.

Replace Invalid SAP Dates with

If an SAP date cannot be converted to a valid date format, the invalid date is converted to the entered value. NULL is supported as a value. When converting the SAP date the two special cases 00000000 and 9999XXXX are checked first.