Data preparation for Visualization Engineer

Visualization engineer walkthrough — consulting ODD tags and metadata to predict BI-tool performance, set dashboard security levels, and prepare data for chart development.

Key words: BI tool, data chart, performance tuning, data preparation, data load mode, dashboard security level, underlying data.

Challenge

I am a Viz Engineer in a team that develops and maintains Tableau dashboards for book orders reporting and ad hoc monitoring of online store performance. I got a task from my stakeholders to introduce a dashboard that tracks digital inventory during high demand (Black Friday, Christmas or Valentine's Day) and stays stable if 5+ people use the report simultaneously. Dashboard is expected to be refreshed near-real-time.

To build the dashboards I have to know the nature of the data since the majority of BI tools do not support complicated data preparation algorithms. I need to spend a lot of time on manual analysis of meta data and basic profiling.

Solution

The ODD Platform tagging system and metadata storage can be consulted during dashboard development to reach better performance and to plan your security standards, e.g. deciding where row-level security based on the user group is needed.

ODD helps you discover what is sensitive and plan the controls; it does not enforce them. ODD tags are a single global directory with no per-tag access control and no user-group binding, so they cannot grant or restrict access by group. Row-level security and group-scoped access are enforced on the data warehouse or BI-tool side — use what you learn in ODD to design those controls there.

Scenario

1. I examine my data source metadata and tags in ODD and find out that it is: - Set to streaming data load mode. - Does not have the KPIs I need but I might be able to calculate them using existing fields. - Has a history depth of ten calendar years. - Contains all sales data, not just the one of customers subscribed to a newsletter. - Has denormalized structure.

Treat the custom-metadata values you read here as informative, not as a validated source of truth. ODD does not type-check metadata values, and a collector-ingested value can be overwritten through the API and then silently reverts on the next ingestion run. Confirm anything load-bearing against the source system. See the custom metadata caveats for the details.

1. I understand that if I establish a connection from Tableau to a data source as-is, a performance will be very poor and definitely won’t satisfy the near-real-time refresh requirement.

2. I decide to introduce the following data preparation steps:

   • I will develop a view that’ll have customers subscribed to a newsletter data only.

   • I will limit the history depth to one day only (run date) as it is an operational, not analytical dashboard.

   • I will select only the fields that I need for my dashboard.

   • I will pre-calculate all KPIs in my view to increase dashboard performance (set calculation resources on DWH side and not on Tableau as BI tools of this type work best with prepared and pre-aggregated data).

Result: I can prevent any collapses during high demand periods when the company requires operational data being reliable and accessible every minute.