Overview

Lookup Processor is a processor component aimed at enhancing a record with a value stored in an external lookup system such as a relational database or redis.

FQCN

com.datumize.ddc.component.lookup.LookupProcessor

Properties


Common properties that apply to all source and processor components of a pipeline.

PROPERTYIDDESCRIPTION

REQUIRED

TYPE

DEFAULT

EXAMPLES

Common
IdentifierIDComponent unique Identifier within the pipeline, read only, only useful for advanced mode.YesStringAuto

MyComponent_23423

This Identifier is automatically generated by the system and you can't change it. Might be helpful for advanced pipeline configuration.

DescriptiondescriptionA short description for the component, to be visualized inside the component representation, aimed at providing additional information to understand the pipeline at a glance.NoString

Extract customer id and loyalty number.

Short and sweet description.

TopictopicAll Source and Processor components support a topic to tag the output records. If the record does not have a tag applied, this topic will be automatically applied. The topic may be used by future components to route, group or classify records.NoString

foo

All output records will be tagged using "foo", unless they have not been tagged during the execution of the step.

PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
Default
Lookup systemLookupLookup system to use.

Yes

See Lookup Systems section



CombinercombinerCombiner to combine the input data with the results of the selected lookup system.YesSee Combiner section

Lookup Systems

DB Lookup

Use a database as lookup system.

For each input record, this processor will look for a row in the database that matches that record.

Note: Due to the query might return multiple rows that match the criteria, DB Lookup will return a list of results.

PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
Default
URLurl

Database URL connection in jdbc format. The actual format depends on the RDBMS.

Batch support on Postgres needs reWriteBatchedInserts=true parameter in URL.

YesURL
jdbc:postgresql://host:port/database
Driver classdriverJDBC driver full class name.YesClassname for the JDBC driver to use.
org.postgresql.Driver
UseruserDatabase user.YesString
datumize
PasswordpasswordDatabase password.YesString
datumize123
Queryquery

SQL Query.

You must define at least one placeholder (?) in the query to filter the content.

YesString
SELECT id, value FROM table WHERE id = ?
Key extractorskey-extractors

Extract the keys from the input object. These keys will replace the '?' placeholders in the query in order.

The number of placeholders (?) in the query must match with the number of key-extractors defined.

Yes

List of Extractors.

See Extractors section.



ConverterconverterThe converter to use in order to convert the resultset to whatever object is the target.YesSee Database Converters converters section.

Connection Pool
Wait timeoutconnection-timeoutMaximum number of milliseconds that a client will wait for a connection from the database connection pool.NoDuration30s

10s

Set connection timeout to 10 seconds.

Idle timeoutidle-timeoutMaximum amount of time that a connection is allowed to sit idle in the database connection pool.NoDuration10s
Lifetime timeoutmax-lifetimeMaximum lifetime of a connection in the poolNoDuration3m
Minimum idleminimum-idleMinimum number of idle connections to maintain in the database connection pool.NoInteger10
Maximum sizemaximum-pool-sizeMaximum number of connections to maintain in the database connection pool.NoInteger10

Redis Lookup

Use Redis as lookup system.

For each input record, this processor will look for a key-value pair in the database of the Redis server that match that record.

Note: Due to it uses key-value pairs, Redis Lookup will return a single result.

PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULTEXAMPLES
Default
Hosthost

Host of the Redis server.

NoStringlocalhost192.168.1.11
PortportPort of the Redis server.NoInteger6379

PasswordpasswordPassword of the Redis server.NoPassword

DatabasedatabaseRedis databaseNoInteger0
Key extractorkey-extractor

Extract a key from the input object.

That key will be used to find in the Redis server a key-value pair that whose key match the key extracted from the object.

YesSee Extractors section

Value deserializervalue-deserializerDeserializer to parse the value returned by the Redis server to the desired data type.NoSee Deserializers section

Advanced
Wait timeouttimeoutMaximum number of milliseconds that a client will wait for a connection with the Redis server.NoDuration2s

10s

Set connection timeout to 10 seconds.

Use SSLsslUse a secure SSL/TLS to establish the connection if the Redis server supports it.NoBooleanfalse