Datumize Overview

Learn some concepts on data, edge and Datumize products. A quick overview to understand the broader picture and how Datumize products fit into an enterprise architecture.

About Datumize

Datumize is a technology vendor established in 2014 in Barcelona working on data integration and edge computing. We are a vendor in the data integration and edge computing space that unlocks data for digital transformation. Datumize captures data, using our unique middleware, optionally embedded in our low-cost industrial hardware, from sophisticated data sources, computes on edge, and ingests into any destination, preserving full data privacy.

We have a novel approach to data capture (passive network sniffing, legacy/proprietary protocols) and edge computing (multi-platform agent, custom business logic, edge hardware). Datumize enables advanced use cases for machine data (legacy equipment to cloud, industrial IoT) and edge computing (remote control, AI on the edge). We have the credibility of world-class clients, leaders in their industry.

You can read more at www.datumize.com

Enterprise Data, Edge Computing and Datumize

The concept of Enterprise Data has enough breadth and depth as to write several books. It is useful to define the high level picture of what enterprise data actually means to most companies and what comprises the life cycle for enterprise data. There is not a single definition for enterprise data or even an enterprise data architecture, so take this classification here as something mostly accepted in the market. Depending on a vendor's alignment, the classification might look different, but you should get the point.

These are well accepted areas of expertise (or capabilities) that any enterprise data architecture should define at some point, as summarized in following table.

These are the usual activities for enterprise data:

• Data Identification: this is the first and probably one of the most neglected activities, aimed at inventorying the data in your organization. Your data inventory should cover what the data describes and what type it is, how the data is generated and estimated volumes, and technical formats (protocol, file type, etc.) You usually find data inventories setup when there is some sort of data governance in place; an inventory, however, can be as simple as a spreadsheet describing the different data sources available in the organization. However, it often happens that data sources are not documented and you need to “read the source code” - that is, having a look into actual real data.

• Data Collection: immediately after identifying your relevant data sources, often heterogeneous and siloed, some collection (or integration) is needed to put together these relevant pieces of data. There are multiple techniques to collect data, including virtual collection that creates a pointer to the data in their original source and status, or physical collection (extract-transform-load et al) in which data is read at source and ingested/sunk into a different place. Often, the data collection includes additional activities such as cleansing, to avoid unnecessary data flow, mapping and format change, to adapt the data to target location needs, and support for multiple protocols, to understand the myriad of data formats that exist in the real world.

• Data Aggregation: collected data tends to be verbose and fine grained, more than fine and mostly mandatory for a detailed and automated/algorithmic analysis; however, human interpretation needs summarization to understand complex concepts, and that means aggregation. The “best sellers” for aggregated data are counting, average, maximum and minimum; most business metrics can be defined in terms of these operations (remember the median). The other usual concern about data aggregations is related to volume and performance: aggregating 1 million data points every 10 seconds is a hard problem.

• Data Analysis: analysis tends to be considered the exciting phase, when advanced mathematics, statistics and business modeling (all of them combined in what is known as data science) unleashes all the expressiveness of data into unexpected (or not) insights and conclusions. Most of the time, analysis starts on a blank sheet of paper and requires a combination of techniques (classical and modern) to create meaningful results. It is well accepted in the market that analytical models can be classified according to a timeline, defining descriptive models (describing the past and current time), predictive models (guessing up the future based on past and present data), and prescriptive models (that involve some sort or knowledge/learning, usually related to Artificial Intelligence, that guess up the recommendations for the future).

• Data Visualization: sight is the dominating sense for most human beings. Therefore, complex information tends to be better transmitted using a visual framework, and data is not an exception to this rule. Data visualization refers to the techniques and tooling used to transmit the metrics, insights and conclusions calculated during the analysis. Infographics, charts, reports or dashboards are examples of techniques used to represent data with varying degree of detail and interactiveness. The concept of a drill-down is one major advantage for visualization tools: it allows humans to cope with summarized views of data and eventually dig into the details for further understanding.

Usually, Edge Computing is not considered as part of the Enterprise Data journey. However, modern architectures and trends in computation are gearing (again) into the edge. Again meaning that history of computation remembers us that data and computation has been moving back and forth between central and distributed locations; so edge computing is not new at all. What might be new, however, is the availability of low-cost computing devices that make this type of distributed processing very affordable for many organizations. The edge contains lots of untapped opportunities for data and computation, that why Datumize provides features to combine data integration and edge computing, including the ability to manage the endpoint, run 3rd party software (eg. containers or AI models) or even custom business logic in scripted languages. 

The following diagram represents how Datumize is positioned in terms of the data journey. Dealing with data actually means to cope with a space between the traditional data integration for Information Technology (IT) and industrial data integration for Operational Technology (OT).

As a product in the data integration space, Datumize main mission is to ingest data into the enterprise data & analytics platform. However, special and sophisticated data sources needed for modern transformational projects, usually mandate to cope with network transactions, Internet of Things (IoT), mobility devices (Wi-Fi, Bluetooth), industrial machines (OPC, Modbus) or highly distributed and disconnected environments.

Data and other types of Data

Data is the raw material for almost every innovation project, but also the reason why many of these projects are failing. According to McKinsey, almost 40% of large corporations are running smart data projects, and other 30% are planning to start one. IBM has reported that 1/3 of business leaders don’t trust the information they use to make decisions. And most surprisingly, 63% of the companies using Big Data have not been successful in data-driven insights, according to Dataversity.

One possible classification for data is made according to 2 factors:

1. Availability, whether the data are stored and made available for further usage. One could argue whether collected and transmitted data in streaming mode is stored or not; for the sake of simplicity, it is assumed that any collected data is stored, even streaming data where some sort of temporal storage (i.e. Kafka) is used.

2. Usage, indicating whether the data is actually used or not for any additional purpose. A typical use would include visualization in a chart.

The table below explains the scenarios when these two factors are considered. This method of data classification is not novel, mostly inspired by the analysis technique of the Johari Window, and widely used in intelligence, engineering and astrophysics. You can read more about the known knowns and related concepts, here.

Gartner defines Dark Data as “information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes.” At least 80% of is dark data; the great hidden resource that flows untapped through major organizations. At Datumize, we define Dark Data as “data not collected and therefore not used.” According to previous data classification, data not collected are considered to be dark data.

Data Sources

Where are most valuable data to be found? According to the data classification proposed previously, considering data against availability and usage factors, Datumize concentrates on gathering data in "not collected" areas, because these areas usually contain sophisticated data that are not straightforward to capture. Datumize products are aimed at data integration and work whenever the data are not being collected; for other cases, such as neglected operational data, there are other data governance and discovery products that should be considered.

The following data sources are found in common scenarios. Please note that the list is not exhaustive.

Network Transactions

Tons of in-transit data remain hidden and unleveraged due to the difficulty of collecting and processing temporary transactions flowing over the network (API, XML integrations, etc.) without the hassle of modifying the backend systems.

Wi-Fi Technology

Wi-Fi, a ubiquitous technology that is present in almost every facility, generates a huge variety of data that remains poorly explored but can be used for delivering motion intelligence to companies.

Industrial Networks

Highly valuable operational data is trapped inside machines, devices and sensors. Vendor lock-in, proprietary protocols and lack of interoperability have inhibited machine data to be shared and used to govern and unlock efficiencies.

Distributed Locations

Unconsolidated IT landscapes with legacy systems and/or heterogeneous technologies are the main reason why the companies still have data silos, that remains a big challenge for corporate intelligence.