Properties are placeholders for a configuration that gets a default value and can be overridden during deployment, providing flexibility for pipelines and supporting enterprise use cases.Properties are highly useful for using and reusing assets, when variations of key fields are anticipated to need changes. This is very useful for fields that require unique names, IDs, passwords, IPs, etc, that will be in a pipeline or deployment but may change between deployments or environments.

Objectives

  • Understand the concept of properties.

  • How to define both Global and component level properties while working with resources.

  • Overriding polices

Overview

Some key concepts are helpful to understand so that a you are able to take advantage of the flexibility and control that is introduced with the Properties editor.

  • A property is a placeholder for a configuration value.

  • By convention, you can call on properties using brackets and the property name, i.e. $${myproperty} for example, when calling a previous set property in a cook processor. For example a use of this feature could also be to set an endpoint as a generic value of $${endpoint}  and later in the DDC pipeline policy using the Add Properties feature to define each policy to a new pipeline copy and assigning unique endpoint values.

  • A property will always have a default value, so if it is not defined this will be used.

  • Properties can be overridden with values defined in a mapping, that is $${myproperty} can have a value of foo in one situation, and lolo in another situation.

  • A property name should be unique, at least within the context of one resource.

Properties are needed to make some resources more generic. A good example is in a DDC pipeline, as you may want to build a pipeline just once, but deploy it in different environments (pre-production, production) where there will be slight changes in the configuration, like server addresses, ports or passwords. Properties  allow you to define those parts of the configuration that are subject to be adjusted.


To see why this is so important and valuable, the following diagram represents the conceptual usage of properties for a typical flow of DDC pipeline building and deployment.

  • In this diagram MyPipeline is a DDC pipeline. The components in green are those that require some adjustment.

  • In this example, the API Source defines two properties, pass1 and url1. Similarly for the API Sink.

  • There is a global property factoryId.

  • The properties are defined and always come with a default value. This default value could be any value, but typically will be a value that can be used in Zentral for debugging purposes.

  • The deployment plan will define the mappings when assigning the pipeline to DDC instances. When deploying MyPipeline to DDC instance DDDpre, in a pre-production environment, the 5 properties will have specific values assigned. Not all properties must be overridden; if you don’t override, you get the default value.

  • The mapping can be defined in the DDC pipeline policy. If it is not defined, default values will be used. If some properties are not needed (old) or properties are missing, default values will be used. In these special cases, however, Zentral will produce a warning.


Defining global properties (at pipeline level)

It is often the case that you may need need to define properties for a broader scope, that is the whole pipeline. Those properties must be defined beforehand, because they represent global identifiers that will be used throughout the rest of the configuration. For example: you are working on a pipeline that generically captures data from SNMP Wi-Fi Access Points, and transforms each read into a JSON document that will be ingested into a certain storage. It is very likely that the JSON document contains a field that should identify the physical location (i.e. a hotel), so to achieve this you will want to use a global identifier that says “this is the Pegasus Hotel, or hotel 123”. 

Global properties are  represented using a complex field that contain at least:

  • Name: an identifier

  • Type: only simple types, such as string or number.

  • Default value: the default value. You always have a default value for each property in case it’s not overridden.


To set global properties on a pipeline, navigate to More Actions:


And then add new properties, configured by clicking on the plus symbol, and then on the cog symbol to reveal the editor for the properties you wish to add.



Defining component properties (at component level)

As you can see, adding a Property in a pipeline component for an appropriate field type, is as easy as setting the tag. First open the component field editor. In the example below, the User field, now has a Property tag, and hovering over the  field for Password enables the property  tag editor (shown with a + ) to become available.


Please keep in mind:

  • The property name must be unique for this pipeline.

  • You can only define one property per field.

  • For field properties, you don’t need to define the type or default value, it can be deduced out of the field.

  • After pressing Enter key it will be assigned to the field value. Then press on Save component and it’ll be saved temporary. However, it won’t be persisted until you Save or Publish your Pipeline.

Note: You cannot create any property with an empty value or with any other component declaration as value (i.e. having a component property with a tag called prop1 and a value $${prop2})


Using properties while working on the pipeline

Properties autocompletion is available by typing $${


Some fields might require you to use properties that are global to the pipeline or global to the product. For example, a script code for Cook Processor will consider properties that have been previously defined.  To use a defined property;

  • When you type $${ the popup dialog opens.

  • You can continue to write, and the properties are automatically filtered

  • If you click on a certain property, the property name is used and } is appended to close the property inclusion (i.e. $${foo}).


Overriding properties in a Deployment Plan

Properties flexibility comes when overriding in a deployment plan.

To override a property the way to do so is to use the DDC Pipeline Policy  in the deployment plan. You will need to publish the previous pipeline version in order to assign it to a new policy.

 In “More actions”  you will find “Properties” that will list all the available properties.

This will open a Properties sidebar with all the global and component properties that were assigned previously in our published pipeline version.

The properties will contain the default value, that can be overridden. This is then assigned to the to a previously created target instance allowing you to reuse a pipeline, but adjusting its policy by including the new property value which you have selected for the new pipeline with only the minor properties having to be changed, and deployed to the machine.This is because when a DDC runs a pipeline, it will always have a DDC Pipeline Policy associated to it:

  • If it’s a default policy, then the properties will not be overridden because the policy does not contain any override for the properties. So the configured value for the property is used without any replacement.

  • If it’s a custom policy, then for each property:

    • If the policy overrides, then DDC will instantiate the pipeline with the overridden value.

    • If it is not overridden, so the configured value is used.