Kafka Destination

Kafka provides a highly scalable and fault-tolerant messaging system that enables real-time data processing and stream processing at scale. When integrated with Segment, Kafka serves as a powerful backbone for managing and processing event data collected by Segment, allowing businesses to efficiently ingest, route, and analyze data across various applications and systems in real time.

Getting started

Create the Kafka Destination

1. From your workspace’s Destination catalog page search for “Kafka”.
2. Select the “Kafka” tile and click Add Destination.
3. Select an existing Source to connect to Kafka.
4. Enter a name for your Kafka destination.

Configure the Kafka Destination

The way you’ve configured your Kafka Cluster informs the authentication and encryption settings you’ll need to apply to the Segment Kafka Destination. You may need the assistance of someone technical to provide values for the following Settings:

5. On the Settings tab, enter values into the Client ID, Brokers and Authentication Mechanism setting fields.

6. Populate fields based on the value you selected from the Authentication Mechanism field:
   • Plain or SCRAM-SHA-256 / 512 authentication: provide values for Username and Password fields.
   • AWS authentication: provide values for AWS Access Key ID and AWS Secret Key fields, and optionally for the AWS Authorization Identity field.
   • Client Certificate authentication: provide values for the SSL Client Key and SSL Client Certificate fields.

7. Populate the SSL Certificate Authority field, if necessary.

8. Save your changes and proceed to Configure the Send Action.

Configure the “Send” Action

9. Select the Mappings tab and add a new Send mapping.

10. Select a Topic to send data to. This field should auto-populate based on the credentials you provided in the Settings tab.

11. Map your payload using the Payload field.
    (Optional): Specify partitioning preferences, Headers and Message Key values.

12. Save and enable the Action, then navigate back to the Kafka destination’s Settings tab to enable and save the Destination.

Destination Settings

Available Actions

Build your own Mappings. Combine supported triggers with the following Kafka-supported actions:

• Send

Send

Send data to a Kafka topic

Send is a Cloud action. The default Trigger is: type = "track" or type = "identify" or type = "page" or type = "screen" or type = "group"

Click to show / hide fields

FAQ

Which Kafka Platforms are supported?

The Kafka Destination can send data to Topics on self-hosted Kafka Clusters, or to Clusters hosted on Managed Service platforms like Confluent Cloud and Aiven.

Which data formats are supported?

Segment sends data to Kafka in JSON format only. Segment does not yet support other formats, like Avro or Protobuf.

Which authentication mechanisms are supported?

The Authentication Mechanism is controlled with the Authentication Mechanism Setting field.

Segment supports the following SASL-based authentication methods:

• Plain
• SCRAM-SHA-256
• SCRAM-SHA-512
• AWS

Segment also supports Client Certificate authentication.

How is partitioning controlled?

The Send Action provides multiple ways to specify which Partition an event should be sent to.

• Partition: Use this field to specify the name of the Partition Segment should send events to.
• Default Partition: Use this field to specify a default Partition. Segment uses this when you don’t provide a value in the Partition field.
• Message Key: Segment uses a hash of this field’s value to determine which Partition should receive an event. If you don’t provide a Message Key, Segment uses a round robin algorithm to select the partition to send the event to.

What is the “SSL - Reject Unauthorized Certificate Authority” field for?

This field specifies if Segment should reject server connections when a certificate is not signed by a trusted Certificate Authority (CA). This can be useful for testing purposes or when using a self-signed certificate.

Engage

You can send computed traits and audiences generated using Engage to this destination as a user property. To learn more about Engage, schedule a demo.

For user-property destinations, an identify call is sent to the destination for each user being added and removed. The property name is the snake_cased version of the audience name, with a true/false value to indicate membership. For example, when a user first completes an order in the last 30 days, Engage sends an Identify call with the property order_completed_last_30days: true. When the user no longer satisfies this condition (for example, it’s been more than 30 days since their last order), Engage sets that value to false.

When you first create an audience, Engage sends an Identify call for every user in that audience. Later audience syncs only send updates for users whose membership has changed since the last sync.

Settings

Segment lets you change these destination settings from the Segment app without having to touch any code.