Kafka Sink
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Sends each message as a record to a topic in Apache Kafka, or in a Kafka-compatible service such as Amazon MSK, Confluent Cloud or Redpanda.
Use it to publish events or data for other applications to read, such as orders, sensor readings or changes from a database. To write the records with a schema kept in a schema registry, such as Confluent Schema Registry, Redpanda or Apicurio Registry, set schemaRegistryVendor.
Input: the body becomes the value of the record, as text. Turn structured data such as a Map into JSON text first, for example with json-serialize-action. With schemaRegistryVendor and schemaFormat json, the body must be JSON text (or a Map), and the record is written with a JSON Schema; with avro, confluent and apicurio write a text body as an Avro string, while aws-glue needs an Avro record. With azure, set schemaGroup. The optional key header (or kafka-key, ce-key, ce-kafkakey) becomes the key of the record, and the optional partition-key header (or ce-partitionkey) is the number of the partition to write to. The other headers of the message, except the Camel ones, are sent as record headers.
Output: the message is unchanged.
Good to know: saslAuthType has no default and must be set; NONE connects with no login and no TLS, as a local Kafka expects. A managed Kafka service needs one of the other types. With schemaRegistryVendor, the integration needs the serde jars of the vendor, which Camel JBang downloads by itself; for confluent add the Confluent Maven repository, as in camel run --repos=confluent. With aws-glue and azure, the serde logs in with the credentials of the environment the integration runs in: the AWS default credentials chain, or the Azure DefaultAzureCredential (such as the AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_CLIENT_SECRET variables). To log in to a confluent or apicurio registry, set schemaRegistryUsername and schemaRegistryPassword (for Confluent Cloud, the API key and secret of the Schema Registry), or the OAuth2 client credentials schemaRegistryOauthClientId, schemaRegistryOauthClientSecret and schemaRegistryOauthTokenEndpoint, such as of a Keycloak client; these options are not used with aws-glue and azure, and setting one fails at startup. Kafka only calls an OAuth2 token endpoint that is listed in the JVM system property org.apache.kafka.sasl.oauthbearer.allowed.urls, such as with camel run --jvm-args=-Dorg.apache.kafka.sasl.oauthbearer.allowed.urls=https://idp.example.com/oauth2/token; without it, the integration fails to start with The URL cannot be accessed due to restrictions. This applies to schemaRegistryOauthTokenEndpoint and to the oauthTokenEndpointUri of saslAuthType OAUTH. The sink registers a schema made from the message, which with json accepts any JSON object; it does not check the message against a schema that is already registered for the topic. Headers that start with CamelKafka, except CamelKafkaKey and CamelKafkaOverrideTopic, are removed from the message first, so they cannot change what the Kamelet does.
Configuration Options
The following table summarizes the configuration options available for the kafka-sink Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Bootstrap Servers | Required The address of one or more Kafka brokers to connect to first, as | string | broker1:9092,broker2:9092 | ||
Authentication Type | Required How to log in to Kafka. It has no default and must be set. Enum values: * NONE * PLAIN * SCRAM_SHA_256 * SCRAM_SHA_512 * SSL * OAUTH * AWS_MSK_IAM * KERBEROS | string | |||
Topic Name | Required The name of the topic to write the records to. | string | orders | ||
OAuth Client ID | OAuth client ID. Required when saslAuthType is OAUTH. | string | |||
OAuth Client Secret | OAuth client secret. Required when saslAuthType is OAUTH. | string | |||
OAuth Scope | OAuth scope. Optional when saslAuthType is OAUTH. | string | |||
OAuth Token Endpoint | OAuth token endpoint URI. Required when saslAuthType is OAUTH. The URL must also be listed in the JVM system property | string | |||
Password | Password for SASL authentication. Required when saslAuthType is PLAIN, SCRAM_SHA_256, or SCRAM_SHA_512. | string | |||
Username | Username for SASL authentication. Required when saslAuthType is PLAIN, SCRAM_SHA_256, or SCRAM_SHA_512. | string | |||
Schema Format | The format of the record values in the schema registry: Enum values: * avro * json | string | avro | ||
Schema Group | The schema group in the Azure Schema Registry that the schemas of the records are registered in. Required when | string | |||
Schema Registry Name | The name of the registry in AWS Glue. When not set, it is | string | |||
Schema Registry OAuth Client ID | The OAuth2 client id, to log in to a | string | |||
Schema Registry OAuth Client Secret | The OAuth2 client secret of | string | |||
Schema Registry OAuth Scope | The OAuth2 scope to ask for when logging in to the schema registry with client credentials (optional). | string | |||
Schema Registry OAuth Token Endpoint | The token endpoint of the OAuth2 server, to log in to a | string | http://keycloak:8080/realms/registry/protocol/openid-connect/token | ||
Schema Registry Password | The password of | string | |||
Schema Registry Region | The AWS region of the AWS Glue Schema Registry, such as | string | |||
Schema Registry URL | The URL of the schema registry: for | string | |||
Schema Registry Username | The username to log in to a | string | |||
Schema Registry Vendor | The schema registry that holds the schemas of the record values: Enum values: * confluent * apicurio * aws-glue * azure | string | |||
SSL Key Password | The password of the private key in the key store file. | string | |||
SSL Keystore Location | The location of the key store file. Used for mTLS authentication. | string | |||
SSL Keystore Password | The password for the key store file. | string | |||
SSL Truststore Location | The location of the trust store file. | string | |||
SSL Truststore Password | The password for the trust store file. | string |
Dependencies
At runtime, the kafka-sink Kamelet relies upon the presence of the following dependencies:
-
camel:jackson3
-
camel:kafka
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:timer-source"
parameters:
period: 10000
message: 'test'
steps:
- to:
uri: "kamelet:kafka-sink" You can now run it directly through the following command
camel run route.yaml Kafka Sink Kamelet Description
Authentication
The saslAuthType option chooses how the Kamelet logs in to Kafka. The options that each type needs are listed below; leave the others empty.
saslAuthType | Options to set |
|---|---|
| None. No login and no TLS, as a local Kafka expects. |
|
|
|
|
|
|
| None in the Kamelet: the AWS credentials are taken from the environment the integration runs in. |
| None in the Kamelet: Kerberos is configured in the environment the integration runs in. |
Schema registry
To write the records with a schema kept in a schema registry, set schemaRegistryVendor, schemaRegistryUrl and schemaFormat (the default is avro). With json, the body must be JSON text (or a Map); the Kamelet turns it into an object, and the record is written with a JSON Schema made from it, which accepts any JSON object. With avro, confluent and apicurio write a text body as an Avro string, while aws-glue needs an Avro record, which a step must create in code. With azure, also set schemaGroup. The sink registers the schema it makes, and does not check the message against a schema that is already registered for the topic.
schemaRegistryVendor | schemaFormat | Registry |
|---|---|---|
|
| Confluent Schema Registry, and the registries with the same API, such as the one of Redpanda and Karapace. |
|
| Apicurio Registry 3.x. |
|
| AWS Glue Schema Registry. Set |
|
| Azure Schema Registry of Event Hubs. |
The serde jars of the vendor are not part of the Kamelet. Camel JBang downloads them by itself; the Confluent jars are in the Confluent Maven repository, not in Maven Central, so run with --repos=confluent. In a Maven project, add the jar of the vendor and format, such as io.confluent:kafka-avro-serializer, and for Confluent the repository https://packages.confluent.io/maven/. To log in to a confluent or apicurio registry, set schemaRegistryUsername and schemaRegistryPassword (basic auth), or schemaRegistryOauthTokenEndpoint, schemaRegistryOauthClientId, schemaRegistryOauthClientSecret and, if needed, schemaRegistryOauthScope (OAuth2). azure and aws-glue log in with the credentials of the environment.
- route:
from:
uri: "kamelet:timer-source"
parameters:
period: 10000
message: '{"id": 42, "item": "book"}'
contentType: application/json
steps:
- to:
uri: "kamelet:kafka-sink"
parameters:
bootstrapServers: "localhost:9092"
topic: orders
saslAuthType: NONE
schemaRegistryVendor: confluent
schemaRegistryUrl: "http://localhost:8081"
schemaFormat: json Usage example
The route below sends a JSON order every ten seconds to the orders topic, with the order number as the record key.
- route:
from:
uri: "kamelet:timer-source"
parameters:
period: 10000
message: '{"id": 42, "item": "book"}'
contentType: application/json
steps:
- setHeader:
name: key
constant: "42"
- to:
uri: "kamelet:kafka-sink"
parameters:
bootstrapServers: "localhost:9092"
topic: orders
saslAuthType: NONE