Overview

File Sink is a sink component aimed at writing records into files in the local file system.

FQCN

com.datumize.ddc.component.file.sink.FileSink

Properties


Common properties that apply to all sink components of a pipeline.

PROPERTYIDDESCRIPTION

REQUIRED

TYPE

DEFAULT

EXAMPLES

Common
IdentifierIDComponent unique Identifier within the pipeline, read only, only useful for advanced mode.YesStringAuto

MyComponent_23423

This Identifier is automatically generated by the system and you can't change it. Might be helpful for advanced pipeline configuration.

DescriptiondescriptionA short description for the component, to be visualized inside the component representation, aimed at providing additional information to understand the pipeline at a glance.NoString

Extract customer id and loyalty number.

Short and sweet description.

PROPERTYIDDESCRIPTIONREQUIREDTYPEDEFAULT
Default
Base directorydirectory-baseBase directory to write files toYesPath
Advanced

Directory pattern

directory-pattern

Directory name pattern to create directories for output files. Useful to implement time partitioning.

No

Filename Pattern

%{year}%{month}%{day}/%{hour}%{minute}

File pattern

file-pattern

File name pattern for files to be created. The combination of file and directory patterns define the partitioning strategy.

No

Filename Pattern

%{uuid}

Closed files suffix

closed-file-suffix

Suffix added to the file when is closed.

No

String


SerializerserializerSerializer to serialize the contentNoSee Serializers section.NoSerializer

Examples

PROPERTYEXAMPLES
Default
Base directory

/var/output

Sets root directory to write output files.

Advanced

Directory pattern

%{year}%{month}%{day}/%{hour}%{minute}

The default directory pattern will create one directory per day and one subdirectory per minute.

output-%{year}%{month}%{day}

This pattern will create directories named such as output-20200215, effectively providing time partitioning for one day.

File pattern

%{uuid}

The default file pattern will create one file per record, using an unique file identifier.

myfile.out

This fixed name is used to dump all records into the file myfile.out. Please note that this file will not be rotated unless a proper directory pattern is in place.

Closed files suffix

.done

When the file being written is just closed, the .done suffix will be appended to the name. This is useful for file synchronization purposes.