Apache Kafka

Overview

Intempt streams records to a Kafka cluster you own. Nothing is hosted for you: you bring the brokers, the credentials and the topics, and Intempt produces messages onto them.

Two pieces have to exist before a single message lands.

The split is deliberate, and the node's own footer states it: broker lists, certificates and SASL credentials live on the connection, not on the node. The node only decides what a message says and where it goes.

📘 Good to know

Topics are never created for you. Whatever a node's topic resolves to has to already exist on the cluster, and it has to be declared on the connection. A workflow publishing to a name that is not there fails rather than inventing the topic.

Before you start

• Bootstrap servers for your cluster, as host:port pairs.
• The topics you want to publish to, already created on the cluster.
• Credentials for the security protocol your cluster uses.
• On SSL or SASL_SSL, the broker's CA certificate as a PEM.

Step 1: connect your Kafka cluster

Go to Integrations, open the Connections tab, click + Add integration, and pick Apache Kafka under Destinations. The panel is titled Configure Apache Kafka.

The panel runs top to bottom through Connection, Security and Topics, then the connection test.

Connection

Security

Security protocol defaults to SASL_SSL, which is what Confluent Cloud and most managed clusters use.

SASL mechanism appears on the two SASL protocols and defaults to PLAIN.

OAUTHBEARER is listed in the dropdown but disabled. It is not supported yet, and it is shown rather than hidden so you find that out before filling in the rest of the form.

What each protocol requires

The panel disables Connect Kafka and names what is still missing until every required field for the chosen protocol is filled.

Bootstrap servers and at least one topic are required on every protocol.

The IAM role ARN has to be a role ARN, not a user ARN. It is a role Intempt assumes to sign the handshake, so no password is stored. It needs kafka-cluster:Connect, WriteData and DescribeTopic on the cluster and on the topics you declare, and its trust policy has to name Intempt's AWS account.

📘 The username is required on protocols that have no SASL at all

Username / API key is required on every security protocol except AWS_MSK_IAM, including PLAINTEXT and SSL. The cluster identifies the connection by it, and a connection saved without one is rejected with username.empty. This surprises people who expect the field to belong to SASL alone.

Certificates

CA certificate is required on SSL and SASL_SSL. It is the broker's certificate authority, so the client can verify the handshake. Paste exactly one certificate: a chain is rejected, because the server reads a single X.509 PEM and would otherwise take whichever came first.

Client certificate is optional, and only for a cluster that requires mutual TLS. It is a separate identity from the username and password above, and either one alone is a complete configuration. The certificate and the Private key are a pair, so supplying one without the other is refused. A Key passphrase is optional. Paste the PEM with or without its BEGIN and END lines.

Both certificate fields read the expiry date out of the PEM and show it back to you, including a warning when the certificate has expired or is close to it. Replacing an expiring certificate does not mean rebuilding the connection: the connection stays, only the certificate changes.

Verify broker hostname is on by default and checks the certificate matches the address you dialled. Turn it off only for a cluster reached through a proxy or tunnel where the names differ by design. With it off the connection is still encrypted, but a broker impersonating the address would be accepted.

📘 Three fields are write-only

The password, the client private key and the key passphrase are stored and never read back. They do not appear in the connection when you reopen it, and they do not appear in a log or an error. You replace them rather than read them. The client certificate itself is a public document and does stay visible.

Topics

List the topics workflows may publish to, one per line. A comma-separated paste works too, and the panel reports how many it counted.

Test connection

Test opens a producer, lists cluster metadata and closes it again. It publishes nothing. On success it reports how many brokers it reached. If the broker rejects the credentials, the panel says so rather than leaving you to find out on the first workflow run.

Click Connect Kafka to save.

Step 2: add the Publish to Kafka node

Open Workflows, open or create a workflow, and add a node. Publish to Kafka is in the Send group.

It takes records from whatever is connected above it: a trigger on all events, a trigger on one named event, or another action. It publishes one message per incoming record.

📘 Good to know

Publish to Kafka is the end of the line. Nothing continues after it, so anything that has to happen as well as the publish goes before it, not after.

Step 3: configure the node

Everything below the connection picker stays greyed out until a connection is selected, because there is nothing to publish to yet.

There is no separate partition key field. A message key is the Message key half set to something other than Nothing, and the broker partitions on it. With no key, the broker partitions round-robin.

If this step fails is not exposed in the panel. Every Publish to Kafka node skips the record and continues, which leaves the run going and records which records were missed.

Choosing what each half publishes

Both halves offer the same three modes.

A message needs a key or a value. Both halves set to Nothing is refused.

Switching a half between Fields and JSON converts rather than starts over, and the templates you typed survive both directions. Going from JSON to Fields asks first when the body is not a flat object, because nested keys become dotted field names and anything that is not a flat value is rewritten as text.

Liquid tags are inserted from the variable picker beside each field, and they are rooted at attribute, as in {{ attribute.user.first_name }}.

Format and schema

Each half declares its own Format, either JSON or Avro. An Avro value beside a JSON key is a normal configuration.

On a JSON half, a Send with schema toggle wraps the value in a schema and payload envelope instead of publishing it bare. It is offered on JSON only, since an Avro half already carries its schema by being Avro.

On an Avro half, leave the Avro schema box empty and the schema is generated from your field names. Paste one and yours is used instead. The panel shows the schema that will actually travel and gives you a Copy for the consumer button.

The node's card on the canvas carries the same four answers the sidebar sets: the topic, what each half publishes and how it is encoded, and what happens on failure. The Message value half sits below the fold of this screenshot, configured the same way.

What lands on the topic

The samples below are real messages from a node publishing an addtocart event, configured with a JSON message value in Fields mode.

Sample message, schema off

With Send with schema off, the value is the flat object your field grid describes:

{
  "brand": "Nimbus",
  "cartId": "CART-186514",
  "category": "Electronics",
  "currency": "USD",
  "eventId": "6117235837195717889",
  "eventName": "addtocart",
  "eventTime": "2026-09-02T09:53:45.626Z",
  "productId": "SKU-11902",
  "productName": "Nimbus Mechanical Keyboard",
  "quantity": "1.0",
  "sourceId": "1497883051600838656",
  "unitPrice": "84.61"
}

Sample message, schema on

With Send with schema on, the same value travels inside an envelope carrying a Kafka Connect style struct schema. The schema is built at publish time from your field names, and every field is typed string and marked optional:

{
  "schema": {
    "type": "struct",
    "optional": false,
    "fields": [
      { "type": "string", "optional": true, "field": "brand" },
      { "type": "string", "optional": true, "field": "cartId" },
      { "type": "string", "optional": true, "field": "category" },
      { "type": "string", "optional": true, "field": "currency" },
      { "type": "string", "optional": true, "field": "eventId" },
      { "type": "string", "optional": true, "field": "eventName" },
      { "type": "string", "optional": true, "field": "eventTime" },
      { "type": "string", "optional": true, "field": "productId" },
      { "type": "string", "optional": true, "field": "productName" },
      { "type": "string", "optional": true, "field": "quantity" },
      { "type": "string", "optional": true, "field": "sourceId" },
      { "type": "string", "optional": true, "field": "unitPrice" }
    ]
  },
  "payload": {
    "brand": "Lumen",
    "cartId": "CART-478605",
    "category": "Home",
    "currency": "GBP",
    "eventId": "5261315822076038361",
    "eventName": "addtocart",
    "eventTime": "2026-09-02T09:50:00.918Z",
    "productId": "SKU-65027",
    "productName": "Lumen Desk Lamp",
    "quantity": "2.0",
    "sourceId": "1497883051600838656",
    "unitPrice": "387.65"
  }
}

The field names in both samples are the ones the author typed into the Fields grid. They are yours to choose, and they are the contract your consumer reads.

📘 Every published value is a string

A Fields half is a map of field name to Liquid template, and a resolved template is text. So "quantity": "1.0" and "unitPrice": "84.61" are strings, not numbers, and the generated schema types every field as string. Cast them in your consumer rather than expecting numeric JSON.

The message envelope

Two rules decide the shape of the whole message, and both matter to a consumer:

• A keyless message has no key field at all. When the Message key half is Nothing, the field is omitted rather than sent as null, and the broker partitions round-robin.
• A valueless message has value set to null. That is the Kafka tombstone, which log compaction reads as "this key is gone". It is only useful with a key set.

Publishing as Avro

Leave the Avro schema box empty on a Fields half and the schema generated from your field names looks like this:

{
  "type": "record",
  "name": "KafkaPublishValue",
  "namespace": "com.intempt.journeys.publish",
  "fields": [
    { "name": "eventName", "type": ["null", "string"], "default": null },
    { "name": "productId", "type": ["null", "string"], "default": null },
    { "name": "unitPrice", "type": ["null", "string"], "default": null }
  ]
}

Field order follows the order you named the fields in, and Avro reads that order as the field order. Both halves use the same record name, so a key schema and a value schema generated this way are both called KafkaPublishValue.

Avro field names have to start with a letter or underscore and contain only letters, digits and underscores. A dotted name produced by flattening a nested JSON body is not a valid Avro field name, and the panel flags it.

Previewing one resolved message

The node's config panel has a One resolved message block showing the whole envelope, topic, key and value together, as one JSON object. Until you pick a profile it shows the templates as written. Pick one and the Liquid resolves against that real person, with any variable the profile has no value for left highlighted.

The schema slot in a Send with schema half is named rather than filled in the preview, because the schema is built by the producer at publish time. Avro is named rather than rendered for the same reason: it is binary, and there is no honest text version of an Avro record.

Here is a value half resolved for a real profile, with Send with schema on. Every template has become a literal, and the schema beside it declares the fourteen fields this node publishes:

The field names are the author's, and the schema follows them. Add a field to the grid and it appears in both halves of this envelope on the next publish.

What the panel refuses to save

The node also refuses to save without a connection and a topic, and the workflow cannot be published while either is unset.

Changing a message that is already publishing

There is no schema registry anywhere in this path, so a consumer decoding with the schema that was live when the workflow started keeps using it. Once a node has published, the panel warns you when an edit would stop publishing a field or change its type.

• Adding a field is safe. A consumer that does not know it ignores it.
• Removing, renaming or retyping a field is not. The consumer breaks inside your own system, and nothing in Intempt fails.

Add a new field rather than renaming the old one, and leave the old one publishing until the consumer has moved.

Use cases

1. Stream every event in a project onto one topic by pairing a trigger that watches all records with a single Publish to Kafka node.
2. Publish a curated subset of an event's attributes instead of the whole record, so the topic carries the contract your consumer wants rather than the shape Intempt happens to store.
3. Key messages by a record identifier so everything about one user stays in partition order and log compaction keeps only the newest.
4. Emit a tombstone for a deleted record by setting the key to the record identifier and the value to Nothing.
5. Publish to a topic derived from the data, using Liquid in the topic field, when the destination topic depends on the record.
6. Turn Send with schema on for consumers built around Kafka Connect, which read the schema and payload envelope directly.
7. Publish Avro to a consumer that already decodes Avro, using the generated schema copied out of the panel.
8. Route different record types to different topics with a multi-split branch and one Publish to Kafka node per output.
9. Mirror an event stream into a data platform your own team owns, without waiting on a batch export window.

Where to go next

• Amazon S3
• Destinations
• Integration catalog
• Source and destination reference
• Event reference