Overview

HTTP API Polling Source is a source component that actively polls a HTTP(S) endpoint whenever activated and injects the response into the pipeline. 

FQCN

com.datumize.ddc.httpapi.component.HTTPApiPollingSource


The DDC pipeline acts as a HTTP client that executes a request and waits for the response. The response is deserialized and converted into a record.

This component must be run just on 1 thread; the batch size is limited to 1 record.

Properties


Common properties that apply to all source and processor components of a pipeline.

PROPERTYIDDESCRIPTION

REQUIRED

TYPE

DEFAULT

EXAMPLES

Common
IdentifierIDComponent unique Identifier within the pipeline, read only, only useful for advanced mode.YesStringAuto

MyComponent_23423

This Identifier is automatically generated by the system and you can't change it. Might be helpful for advanced pipeline configuration.

DescriptiondescriptionA short description for the component, to be visualized inside the component representation, aimed at providing additional information to understand the pipeline at a glance.NoString

Extract customer id and loyalty number.

Short and sweet description.

TopictopicAll Source and Processor components support a topic to tag the output records. If the record does not have a tag applied, this topic will be automatically applied. The topic may be used by future components to route, group or classify records.NoString

foo

All output records will be tagged using "foo", unless they have not been tagged during the execution of the step.

PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
Default
Authenticationauth-methodAuthentication methodYesAuthentication Specifications

Requestrequest-biulderRequest specificationYesRequest Specifications

Advanced
DeserializerdeserializerDeserializers the body responseNoDeserializer
JSONMapDeserializer

Responseresponse-checkerChecks is response is valid or not

No

Response Checkers
Is200HttpCodeResponseChecker

Timeout

timeout

Connection timeout.

No

Duration

10s

Wait 10 seconds until aborting connection while waiting for response.

Application protocolsapplication-protocols

Available protocols for the ALPN (Application-Layer Protocol Negotiation) extension of TLS

NoList


Skip domain verificationskip-domain-verificationWhether or not the certificate domain verification should be skippedNoBooleanfalse

Request Builder Specifications

Request builders are the components responsible to build the HTTP request defining URL, path, body (if any), headers....

GetRequest

Builder to create a GET request .

PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
URL builderurl-builderBuilds the URL for DDC to communicate withYesURLBuilder


HeadersheadersCustom headers to send with the requestNoMap
x-business-header=acme;x-another-header=foo

ModifierRequest

Builder to create a POST, PUT or PATCH request .

PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
URL builderurl-builderBuilds the URL for DDC to communicate withYesURLBuilder


VerbverbVerb or HTTP method YesString
POST
HeadersheadersCustom headers to send with the requestNoMap
x-business-header=acme;x-another-header=foo
SerializerserializerSerializer to convert the record to the HTTP bodyNoSerializer
MapJsonSerializer
When using the ModifierRequest is mandatory to specify the HTTP Content-Type header inside the headers property.

ScriptRequest

Totally customizable request builder. 

PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
ScriptscriptScript that MUST return a SimpleHttpRequestYesString

LanguagelanguageScript languageNoStringgroovy
ParametersparametersCustom parameters to pass to the scriptNoMap

param1=2;param2=ACME

DependenciesdependenciesPath to external jars that will be added to the classloaderNoList of Paths
/home/acme/coyote.jar
Bootstrap scriptscript-bootstrapScript to execute when the step is initialized. Useful to initialize classesNoString

URLBuilder

Simple URL Builder

Simple builder to create urls.

PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
urlurlEndpointyesString

https://acme.com/api/machines/

Script URL Builder

Totally customizable url builder.

PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
ScriptscriptScript that MUST return a SimpleHttpRequestYesString

LanguagelanguageScript languageNoStringgroovy
ParametersparametersCustom parameters to pass to the scriptNoMap

param1=2;param2=ACME

DependenciesdependenciesPath to external jars that will be added to the classloaderNoList<Path>
/home/acme/coyote.jar
Bootstrap scriptscript-bootstrapScript to execute when the step is initialized. Useful to initialize classesNoString

Authentication Specifications

The following specifications are provided.

Basic Authentication (HTTP auth basic)

Implementation to support HTTP Basic Authentication.

PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
UseruserUser name.YesString
datumize
PasswordpasswordPassword.YesString
datumize123

Cookie Authentication (HTTP auth cookie)

Implementation to support server-side authentication based on cookies.

PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
UseruserUser name.YesString
datumize
PasswordpasswordPassword.YesString
datumize123
Login URLlogin-endpointLogin endpoint.YesURL
http://foo:8080/login
Request payloadlogin-bodyLogin request body. User and password must be defined as {user} and {password}.YesString

Request content typelogin-content-typeLogin request content type.NoStringapplication/x-www-form-urlencoded
Login TTLlogin-TTLTime between re-loginsNoDuration5m

X509 certificate Authentication (HTTP auth x509)

Implementation to support certificates

PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
CertificatecertificateContent of the certificateYesString

Private keyprivate-keyContent of the private keyYesString

Algorithmkey-algorithmAlgorithm used to sign the keyNoStringRSA

Kerberos HTTP Authentication

PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
UseruserUser name.YesString
datumize
PasswordpasswordPassword.YesString
datumize123
Krb5krb5Kerberos configuration file.YesString
/etc/krb5.conf
/etc/krb5.conf
LoginloginLogin configuration file.YesString
login.conf
login.conf

Response Checker Specifications

Response checkers are responsible for deciding if a response is valid or not

Is200HttpCodeResponseChecker

Response is considered valid if HTTP status code is 2xx

SimpleHttpCodeResponseChecker

Response is considered valid if HTTP status code is the provided value, error otherwise


PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
Codeok-codeValid response if HTTP status code is the provided value, error otherwiseNoInteger200

204

ScriptResponseChecker

Fully customized checker via script

PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
scriptscript

Script that MUST throw an exception if response is not valid, void if it is fine.

If the response is empty the script input parameter is null. Otherwise, the input map contains:

http-code: HTTP code status

http-protocol: HTTP protocol in the HTTP response status line

reason: HTTP reason in the HTTP response status line

body: if HTTP response has body and deserializer is defined, the deserialized value, the raw bytes otherwise

YesString

LanguagelanguageScript languageNoStringgroovy
ParametersparametersCustom parameters to pass to the scriptNoMap

param1=2;param2=ACME

DependenciesdependenciesPath to external jars that will be added to the classloaderNoList<Path>
/home/acme/coyote.jar
Bootstrap scriptscript-bootstrapScript to execute when the step is initialized. Useful to initialize classesNoString

DeserializerdeserializerDeserializer for the response bodyNoDeserializer