Documentation
Developer Guide
Current: Source Components

Source Components

Source components are aimed at capturing data from the world outside DDC boundaries, do some processing on the data using Data Converters, and create one or several records that are handed over to the Streams and Records that connects the source component to the rest of the pipeline. Usually, you would expect to capture data from a file, databases, REST call or network traffic.

Cook Source — The Cook source offers the possibility to script in the source component directly, in any of the supported languages but with Groovy being the most flexible, so that new and dynamic expansion of the pipeline can be achieved such as calling external libraries or system commands

Database Source — Database Source is a source component aimed at accessing a JDBC compliant database, execute a query and convert results into records to be processed in the pipeline.

File Chunk Source — File Chunk Source is a source component aimed at efficiently reading really large files in chunks. It extends the functionality of a File Source component but adds additional features to deal with chunking.

File PCAP Source — The File PCAP Source is a source component aimed at efficiently reading PCAP files. It extends the functionality of a File Source component but adds additional features to deal with network traffic recorded in PCAP format. It does not support custom deserializers because it is focused on deserializing PCAP format to PCAP4J packets.

File Source — File Source is a source component aimed at picking up files from a directory to be processed in the pipeline.

Hello Source — Hello Source is a sample source component that produces a batch of "Hello World" strings and is useful for learning purposes.

HTTP API Listening Source — HTTP API Source is a source component that opens a HTTP(S) endpoint and listens to incoming connections.

HTTP API Polling Source — HTTP API Polling Source is a source component that actively polls a HTTP(S) endpoint whenever activated and injects the response into the pipeline.

Kafka Source — Kafka Source is a source component aimed at consuming data from a Kafka https://kafka.apache.org topic, deserializing the data and injecting a record into the pipeline.

MQTT Source — The MQTT source listens for messages being sent through a MQTT broker and can support multiple topics.

OPC DA Client Source — Open Platform Communications (OPC) is a series of standards and specifications for industrial telecommunication. OPC Client Source is a polling source component aimed at accessing an OPC DA server, subscribe to one or multiple items and retrieve the status update of these items.

Serial Port Source — Serial Port Source is a source component aimed at consuming data from a serial port usually implementing RS-232 standard.