Shift, Shape & Shepherd your Data

We see organizations as entities that shift, shape, and shepherd the data they ingest, generate, and push out. Let us explain.

Organizations of all sizes and types (academic, for profit, government, or non-profit) operate within a massive "data landscape". And each organization needs to address the increasing "datafication" of the processes that deliver value to the organization, where most processes are both data-centric and data-heavy.

Organizations want their data to deliver actionable insights and to accelerate the organization’s progress towards its desired outcomes. Currently that’s not a reality for all organizations. We want to help.

We’ll use the following diagram to describe our vision:

In the horizontal axis we show Stakeholders: External on the left side, Internal on the right side.
In the vertical axis we show Inputs at the bottom, and Outputs at the top.
In the middle of the diagram are the 03 core data-handling processes each organization needs:

Shift – Taking data from one place or process and moving data to a different place or process.
Shape – Transforming data from one format / standard to another to accelerate intended usage.
Shepherd – Enforcing that data-shipping and data-shaping rules are applied widely and consistently.

The flow of data inside an organization has 04 phases, listed below.

Phase 01: Data Ingestion

Data comes in from outside sources in different formats, using different (or none) standards.

Outside sources of data include: customers, suppliers, as well as details on customer expectations.

In this phase the organization needs to:

first “shift” the external data from the sources to some type of internal repository.
then have repeatable processes to “shape” the incoming external data into formats / standards the organization’s internal processes can consume

Phase 02: Internal Data Consumption

Data is both consumed and generated internally during the operations of the organization.

The external data, once transformed into an internal standard representation, is incorporated into the organization’s internal processes.

In this phase the organization needs to:

first “shift” internally-generated data across the organization
then have repeatable processes to “shape” all internally-produced data (across different functional areas) into an organization-wide common representation

Phase 03: Data Materialization

Data is materialized by the organization into products and/or services.

The aggregate data managed by the organization is used to create saleable products and/or services.

In this phase the organization needs to:

first “shift” frozen-in-time subsets of relevant internal data into permanent, version-controlled storage
secondly “shape” that relevant data into well-defined outcomes (products / services)

“Relevant data” refers to a subset of all internal data that actually goes into building and delivering the organization’s products and/or services.

Phase 04: Data Publishing

Data has to be submitted to outside entities in different formats as well to meet third parties’ requirements. Outside destinations of data include: regulators, customers, suppliers, and consumers.

In this phase the organization needs to:

first “shift” the product-specific or service-specific data to outside stakeholders
and such data sent externally has to be “shaped” to meet the requirements of outside stakeholders

Based on the above phases, an organization can be perceived as a data-shifting and data-shaping hub:

'data shifting' by routing external data in, and internal data out
'data shaping' by taking all kinds of data (both externally-sourced as well as internally-generated) and transforming such raw data into specific formats the organization can digest and eventually export.

The Data Shaping, Shifting, and Shepherding Organization

The success of most organizations is heavily impacted by how they manage their data.

In most cases, staffers already have lots of data available to them. What’s missing is a repeatable process to classify and organize the data to deliver actionable insights. The ultimate goal of all data management activities is to accelerate the organization’s progress towards their desired outcomes.

Data as Water

It’s a well-known challenge that most organizations are “drowning in data” while trying to extract meaning from such data.

The mental representation of “data” in most staffers’ minds uses concepts such as:

river - “a flood of data”
waterfall / cascade
“like drinking from a firehose”

It’s interesting to note the remarkable convergence towards the concept of water, or liquid, with the corresponding concept of “amorphous” (a liquid has no intrinsic shape).

Data Shaping

Organizations need to give data a “shape” by describing the “buckets” or “channels” where amorphous data is either stored in or flows throw.
Organizations also need to enforce such definitions across the organization.
Using a visual representation: a hydroelectric dam turns accumulated water into electrical power.
By channeling (“shaping”) amorphous water down a specific path to make the electricity-generating turbines rotate.
How to “shape” the data so that it generates “organizational power” is determined by standards and taxonomies.

Data Shifting

A maritime port receives ships from an endless number of sources. Carrying endless types of cargo. On board a large number of ship names from a large list of possible ship lines.
Once unloaded, the cargo needs to be routed to an endless number of destinations. By either truck or railroad. And the operations work in both directions: unloading cargo from ships, and then loading cargo on to ships.
Interestingly enough, the port operator does not need to own neither ships, cargo, trucks, railroads, nor the cargo moving through the port at all.
The port operator is basically “sorting and shifting” cargo, in both directions.
An organization is similarly a “data sorting and shifting” entity.

Data Shepherd

A railroad network serves not only as a “sorting and shifting” mechanism to route both people and cargo across a geographical domain. But also takes a “shepherd” role of conducting all stakeholders (from shippers to passengers) to their destinations based on pre-defined schedules.
Either in a local area (a city’s local transport network) or across longer distances (like the German ICE railroad network diagram on the left).
Note: a railroad analogy may be easier for the average person to understand (based on their daily interactions with local transit systems), as opposed to port operations.
Organizations need to then “shepherd” or enforce the use of approved standards and terminologies across all processes, from external data ingestion to external data distribution.

Contact us

Please contact us if your organization has a data management challenge you need assistance with.