Overview

Kafka Source is a source component aimed at consuming data from a Kafka topic, deserializing the data and injecting a record into the pipeline.

FQCN

com.datumize.ddc.component.kafka.KafkaSource


This source component uses a Kafka client and connects to a Kafka topic that contains some data to consume. When the Kafka source component is activated, it connects to a Kafka topicFor each Kafka message retrieved, selected deserializers are applied and an output record is created. The connection supports advanced parameters.

Properties


Common properties that apply to all source and processor components of a pipeline.

PROPERTYIDDESCRIPTION

REQUIRED

TYPE

DEFAULT

EXAMPLES

Common
IdentifierIDComponent unique Identifier within the pipeline, read only, only useful for advanced mode.YesStringAuto

MyComponent_23423

This Identifier is automatically generated by the system and you can't change it. Might be helpful for advanced pipeline configuration.

DescriptiondescriptionA short description for the component, to be visualized inside the component representation, aimed at providing additional information to understand the pipeline at a glance.NoString

Extract customer id and loyalty number.

Short and sweet description.

TopictopicAll Source and Processor components support a topic to tag the output records. If the record does not have a tag applied, this topic will be automatically applied. The topic may be used by future components to route, group or classify records.NoString

foo

All output records will be tagged using "foo", unless they have not been tagged during the execution of the step.


PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
Default
Kafka URLurlOne of several Kafka connection URL, separated by comma.YesURL List

myserver:3456,otherserver:3434

A list of valid Kafka URL separated by comma.

Kafka Topickafka-topicKafka topic to consume data from.YesString

customers

Connect to a Kafka topic named "customers".

Key deserializerkey-deserializerDeserializer to process the consumed key.NoSee DeserializersNo deserializationSee the list of supported Deserializers
Value deserializervalue-deserializerDeserializer to process the consumed value.YesSee DeserializersNo deserializationSee the list of supported Deserializers
Authenticationkafka-authAuthentication details.NoSee Authenticaton methodsNo AuthSee the list of supported Authentication methods
Advanced
Client Idclient-idIdentifier for the Kafka client.NoStringRandom UUID
Group Idgroup-idIdentifier for the group of Kafka consumer processes to which this consumer belongs.NoStringRandom UUID
Polling timeoutpolling-timeoutTimeout for consumer polling operation.NoDuration10s
Auto commitauto-commitEnable auto commit.NoBooleantrue
Auto commit intervalauto-commit-intervalKafka auto commit interval.NoDuration100ms
Advanced parametersadvanced-parametersKafka advanced client parameters, according to Kafka documentationNoJSON List 
{"key1":"value1","key2":"value2"}


Authentication methods

Kafka allows using different authentication mechanism to verify the identity of users and applications.

DDC supports Simple Authentication and Security Layer (SASL) and mutual TLS (mTLS) using X.509 certificates.

No Auth

It establish an unauthenticated connection with the Kafka broker.

SALS

It uses Simple Authentication and Security Layer (SASL) and can be enabled concurrently with TLS encryption.

DDC only supports PLAIN authentication with or without TLS:

  • PLAINTEXT: Un-authenticated, non-encrypted channel.
  • SSL: SSL channel.
  • SALS_PLAINTEXT: SASL authenticated, non-encrypted channel.
  • SASL_SSL: SASL authenticated, SSL channel
PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
Default
Security protocolsecurity-protocolSASL security protocol.No

PLAINTEXT

SSL

SALS_PLAINTEXT

SASL_SSL



UseruserUsername.YesString

PasswordpasswordPassword.YesString

X509

It uses SSL (TLS) client authentication through X.509 private key / certificate pairs.

PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
Default
CertificatecertificateContent of the client's certificate.Yes

String



Private Keycertificate-private-keyContent of the client's private key.YesString

CA certificateca-certificateContent of a trusted certificate of a Certification Authority (CA) with which the client's certificate has been signed.NoString

Advanced
Key Algorithmkey-type-algorithmAlgorithm used to generate the key pair.NoStringRSA
TLS protocoltls-protocolSSL/TLS protocol used to generate the SSL context. See ssl.protocolNoStringTLSv1.3

TLSv1.2

TLSv1.3

Enabled protocolsenabled-protocolsThe list of protocols enabled for SSL/TLS connections. See ssl.enabled.protocols NoList
TLSv1.2, TLSv1.3
Endpoint id algorithmendpoint-id-algorithmThe endpoint identification algorithm to validate server hostname using server certificate. See ssl.endpoint.identification.algorithmNoStringhttps
Key manager algorithmkey-manager-algorithmThe algorithm used by key manager factory for SSL/TLS connections. See ssl.keymanager.algorithmNoStringSunX509
Trust manager algorithmtrust-manager-algorithmThe algorithm used by trust manager factory for SSL/TLS connections. See ssl.trustmanager.algorithmNoStringSunX509