Kafka Batch Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Receives records from one or more topics in Apache Kafka in batches, as one message that holds a list of up to batchSize records.
Use it to handle many records at once, for example to write them to a database in one go, or to save the position in Kafka only once a whole batch was handled. To handle one record at a time use kafka-source. When the records are written with a schema kept in a schema registry, such as Confluent Schema Registry, Redpanda or Apicurio Registry, set schemaRegistryVendor.
Output: the body is a List with one entry per record, up to batchSize. Each entry is a Camel Exchange for one record, not a message: a step reads the value of the record, as text, with getMessage().getBody() on the entry, such as ${body[0].getMessage().getBody()}. With schemaRegistryVendor, each value is read with its schema from the registry and is JSON text, such as {"id": 7, "item": "camel"} for an Avro record; a record without a value (a tombstone) gives an entry with no body. The message of the entry also has the key, offset and timestamp of the record in its CamelKafkaKey, CamelKafkaOffset and CamelKafkaTimestamp headers, and the record headers, as bytes. The message itself has the CamelKafkaTopic and CamelKafkaPartition headers. The kafka-topic, kafka-key and other headers of kafka-source are not set.
Good to know: saslAuthType has no default and must be set; NONE connects with no login and no TLS. To save the position in Kafka only once a batch was handled, set allowManualCommit to true and autoCommitEnable to false, and end the flow with kafka-batch-manual-commit-action. A new consumer group starts at the newest records, unless autoOffsetReset is earliest. Without consumerGroup, every start uses a new group, so the records sent while the integration was stopped are skipped. With schemaRegistryVendor, the integration needs the serde jars of the vendor, which Camel JBang downloads by itself; for confluent add the Confluent Maven repository, as in camel run --repos=confluent. With aws-glue and azure, the serde logs in with the credentials of the environment the integration runs in: the AWS default credentials chain, or the Azure DefaultAzureCredential (such as the AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_CLIENT_SECRET variables). To log in to a confluent or apicurio registry, set schemaRegistryUsername and schemaRegistryPassword (for Confluent Cloud, the API key and secret of the Schema Registry), or the OAuth2 client credentials schemaRegistryOauthClientId, schemaRegistryOauthClientSecret and schemaRegistryOauthTokenEndpoint, such as of a Keycloak client; these options are not used with aws-glue and azure, and setting one fails at startup. Kafka only calls an OAuth2 token endpoint that is listed in the JVM system property org.apache.kafka.sasl.oauthbearer.allowed.urls, such as with camel run --jvm-args=-Dorg.apache.kafka.sasl.oauthbearer.allowed.urls=https://idp.example.com/oauth2/token; without it, the integration fails to start with The URL cannot be accessed due to restrictions. This applies to schemaRegistryOauthTokenEndpoint and to the oauthTokenEndpointUri of saslAuthType OAUTH.
Configuration Options
The following table summarizes the configuration options available for the kafka-batch-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Bootstrap Servers | Required The address of one or more Kafka brokers to connect to first, as | string | broker1:9092,broker2:9092 | ||
Authentication Type | Required How to log in to Kafka. It has no default and must be set. Enum values: * NONE * PLAIN * SCRAM_SHA_256 * SCRAM_SHA_512 * SSL * OAUTH * AWS_MSK_IAM * KERBEROS | string | |||
Topic Names | Required The topics to read from, separated by commas. When | string | orders | ||
Allow Manual Commit | If true, each message carries what | boolean | false | ||
Auto Commit | If true, the position reached in each partition (the offset) is saved in Kafka automatically every 5 seconds, so that a restarted integration goes on from there. Set it to false together with | boolean | true | ||
Auto Offset Reset | Where to start when the consumer group has no saved position yet: | string | latest | earliest | |
Batch Size | The largest number of records in one message. A message holds fewer when fewer records are waiting. | integer | 500 | ||
Batching Interval | When set, a batch that has not reached | integer | 10000 | ||
Consumer Group | The name of the consumer group. Integrations with the same group share the records of the topics between them, and the group remembers the position reached, so that a restarted integration goes on where it stopped. When it is not set, a new random group is used on every start. | string | my-group-id | ||
Automatically Deserialize Headers | Meant to turn the values of the record headers into text. In a batch it has no effect: the record headers of the entries of the list stay bytes. | boolean | true | ||
Max Poll Interval | The longest time, in milliseconds, that the integration may take to handle one batch. When it takes longer, Kafka considers it failed and gives its partitions to another member of the consumer group. When it is not set, the Kafka default of 300000 (5 minutes) applies. | integer | 600000 | ||
OAuth Client ID | The ID of the client registered with the OAuth 2.0 provider. Required when | string | |||
OAuth Client Secret | The secret of the OAuth client | string | |||
OAuth Scope | The scope to ask the OAuth 2.0 provider for, when it requires one. Only used when | string | kafka | ||
OAuth Token URL | The URL of the OAuth 2.0 provider where the Kamelet gets its access token. Required when | string | https://auth.example.com/oauth2/token | ||
Poll On Error Behavior | What to do when reading from Kafka fails: Enum values: * DISCARD * ERROR_HANDLER * RECONNECT * RETRY * STOP | string | ERROR_HANDLER | ||
Poll Timeout | How long one read from Kafka waits for new records, in milliseconds. | integer | 5000 | ||
Password | The password of | string | |||
Username | The user name to log in to Kafka with. Required when | string | |||
Schema Format | The format of the record values in the schema registry: Enum values: * avro * json | string | avro | ||
Schema Registry Name | The name of the registry in AWS Glue. When not set, it is | string | |||
Schema Registry OAuth Client ID | The OAuth2 client id, to log in to a | string | |||
Schema Registry OAuth Client Secret | The OAuth2 client secret of | string | |||
Schema Registry OAuth Scope | The OAuth2 scope to ask for when logging in to the schema registry with client credentials (optional). | string | |||
Schema Registry OAuth Token Endpoint | The token endpoint of the OAuth2 server, to log in to a | string | http://keycloak:8080/realms/registry/protocol/openid-connect/token | ||
Schema Registry Password | The password of | string | |||
Schema Registry Region | The AWS region of the AWS Glue Schema Registry, such as | string | |||
Schema Registry URL | The URL of the schema registry: for | string | |||
Schema Registry Username | The username to log in to a | string | |||
Schema Registry Vendor | The schema registry that holds the schemas of the record values: Enum values: * confluent * apicurio * aws-glue * azure | string | |||
SSL Key Password | The password of the private key in the key store file in | string | |||
SSL Keystore Location | The path of a key store file, in JKS format, with the certificate and private key of the client. Needed only to log in with a client certificate (mutual TLS), usually with | string | /etc/kafka/keystore.jks | ||
SSL Keystore Password | The password of the key store file in | string | |||
SSL Truststore Location | The path of a trust store file, in JKS format, with the certificates to trust the brokers by. Needed only when the brokers use a certificate that Java does not trust by default, such as a self-signed one. | string | /etc/kafka/truststore.jks | ||
SSL Truststore Password | The password of the trust store file in | string | |||
Topic Is Pattern | If true, | boolean | false |
Dependencies
At runtime, the kafka-batch-source Kamelet relies upon the presence of the following dependencies:
-
camel:jackson3
-
camel:kafka
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:kafka-batch-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml Kafka-batch-source Kamelet Description
Authentication methods
The saslAuthType option chooses how the Kamelet logs in to Kafka, as for kafka-source. It has no default and must be set; NONE connects with no login and no TLS, as a local Kafka expects.
Output format
Each message holds a batch: its body is a List with one entry per record, up to batchSize. Each entry has the value of the record as its body, and the record’s key, offset, timestamp and headers as its headers. To save the position in Kafka only once a batch was handled, set allowManualCommit to true and autoCommitEnable to false, and end the flow with kafka-batch-manual-commit-action.
Configuration
The Kamelet requires Kafka connection parameters:
-
topic: The Kafka topic to consume from -
bootstrapServers: Comma separated list of Kafka Broker URLs -
Security-specific parameters based on the authentication method
Usage example
apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
name: kafka-batch-source-binding
spec:
source:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1alpha1
name: kafka-batch-source
properties:
topic: "my-topic"
bootstrapServers: "kafka-broker1:9092,kafka-broker2:9092"
sink:
ref:
kind: Service
apiVersion: v1
name: my-service