Enterprise Integration Summit

Integration Powers Digital Transformation for APIs, Apps, Data & Cloud

REGISTER

March 25, 2021

An Online Conference

The pace of digitization is accelerating. The volume, velocity, and variety of data is exploding.

Customers, clients, partners, and programs are screaming for more data, faster. Central IT is under more pressure than ever, but our budgets are neutral, so we have to do more with less. At times, it can feel like we’re trapped in a perpetually spinning wheel with no destination.

But there is a way out:

The answer is the democratization of data through empowerment and advanced automation strategies. When secure, relevant data is readily available to all of our users, we can build a culture of data self-reliance, get more from our existing data resources, and promote innovation across an organization.  

These are the ingredients of a new recipe for agile and responsive self-serve analytics.

The Future of Self-Serve Data Analytics is Now

A business user needs to have confidence in the data they are using. Implementation of solutions that automatically assess the health, relevance, and usefulness of the data and point out any problems, provide an easy way to remedy any trust issues.

Data preparation is a vital step that will ensure your data is accurate, of the required quality, and useful for specific situations including machine learning. For example, if a non-business user is combining data from two different sources and they are different formats, errors and analysis problems can arise.

Automated, intelligent self-serve analytics solutions recognize and fix issues, so the organization’s data is useful for everyone involved in the environment.

Today’s modern data lakes and complementary technologies for data integration and governance are also working to set users up for self-service success. They help empower business users to easily gain access and analyze data in a cloud data warehouse from either an on prem or a SaaS application. This is where a number of non-tech user focused data ingestion tools come in. Then, your next step is to get out of the way.

Let’s explore possibilities…

Nimble and Safe Empowerment and Data Literacy

CIOs are responsible for satisfying the data demands of every department across the organization. Their customers don't want them to be a bottleneck just as much as CIOs don't want to be the bottleneck; they're just trying to solve problems.

Some of those problems require a lot of data analysis, but there are many narrower problems that only require a subset. It just doesn’t make sense to treat every problem the same way.

Central IT is a necessary resource for questions that demand a lot of analysis. But it’s often delivering an overpowered solution for simpler problems with smaller, department-specific datasets. We need to do something different to accelerate the process — a newer methodology to liberate our customers.

Most lines of business have somebody who is data literate. They're usually not technologists, but they have enough technical skill to build a complicated spreadsheet or run some basic SQL. Let’s look at some modern solutions that can leverage the data-savviness of many of today’s knowledge worker – without all the IT over-engineering.

Nimble datasets: New data lake technology makes it possible to empower these users to get the data for themselves. They are already familiar with the business, and they understand the data. Users can provide a copy of a dataset so that they can help shape the data and improve its quality of the data by doing simple tasks.

These ‘nimble datasets’ don’t require a very high degree of technical skill or time. Most importantly, they don't need to bring a developer into the picture. With modern data lakes, the cost of liberating data is surprisingly low. You can give users a view of the data, instead of creating a full duplicate.

Best of all, users can still hide sensitive or personal information and expose only what is necessary, so you can comply with all the governance and policy rules. And when users  create a “copy” in modern technologies, you don't consume a whole new infrastructure or double your data storage.

User-Friendly Data Catalogues: Another tactic is to provide a data catalog which provides an inventory of all the data that is available within a company. It essentially maintains all the metadata that describes the data and shows users that have access to the catalog what data is available, where and how to find it, and how to use it. In some cases, there are also some additional capabilities that allow users to put their own tags, ratings, etc. on a dataset to give further information on the relevance and usefulness of the dataset.

Data catalogs have traditionally been used by IT organizations. That said, as they are more and more being made available to non-IT business users, they also need to become more user friendly and easier to use by these non-tech users.

The Most Expensive Call You Didn’t Make

Rapid access to information changes the way we make decisions.

For example, say you wanted to add more headcount to your SMB segment. You don’t have the data, but your gut tells you that this segment has potential. Suddenly you get an SMB segment report that tells you, “Oh, by the way, the southeast quadrant is doing really well.”

You would never have seen that if you didn't have the right breakdowns, but now you won’t waste resources on a region where there’s already a natural demand and can instead invest more heavily in the regions that need help. You may have spread headcount equally in all regions, rather than by demand, if you didn't have insight at this level of detail.

We make such decisions all the time, but we are not 100 percent data-driven by any means. As we receive more data, we become better and better at assessing it. Once you see the data clearly, you will find more opportunities. You always do.

The Future is Calling, Time To Act

The transition and transformation to user self-sufficiency is just beginning its a journey, as they say. There's plenty of opportunity for growth and innovation. What’s exciting about this evolution is that it does not preclude central IT from doing all the right things in the right way for the future.

CIOs can empower departments to take care of their business and still do everything they need to do for the long-term health of their data. The secret to success is to get away from ‘central planner’ thinking.  Central IT doesn’t have to be the source of all innovations for your data. In fact, there are benefits to a more de-centralized approach.

For instance, when you so much data at the central level, you have to worry about every action. One wrong step can compromise the business by exposing the data to the wrong people.

But now, IT doesn’t have to design and deploy everything. IT is now empowered to  enforce policy throughout the process.

It works like this:

IT continues to own and configures the storage and the computing environments. Meanwhile, The team prescribes the framework for integrating new data in a way that everyone can follow. They establish policy and standardize tools every step of the way, from collection to storage, to wrangling, to analysis. With these methods and models in place for all the standard data processing, IT just has to deal with exceptions, not every request.

As humans, we have our failings, and by bringing more humans into the picture you could multiply the failings, as well. But with more people doing more things, any one mistake is more likely to be caught and less likely to be catastrophic. And when you work with data constantly, you get better at it.

By empowering our customers with self-service access to the data they need, we free up IT to think about higher-level, higher-order functions that provide so much value and facilitate data literacy. We can anticipate the next big thing and gain insight with an incredible amount of detail instead of reacting to the last one — and, more importantly, help our customers anticipate.

This is the future. Our customers demand it, our performance requires it, and our success depends upon it. The time has come to wake up and solve the need.

As CTO at Talend, Krishna Tammana is responsible for scaling the company’s product and engineering organizations. Previously, Krishna spent nearly a decade as VP of Engineering at Splunk where he led global engineering teams and cloud operations during the company’s successful portfolio expansion and transition to the cloud.