Data driven decision making with distributed information

Artificial Intelligence concept. Creative brain concept background

There is no shortage of information in today’s digital world. But frequently, the information you need to undertake a task is spread across multiple locations, in many repositories, and owned by different teams; and you will often need to use different tools to access all the data you need.  To do your task properly, you need access to all the right information, regardless of where its stored; and you may also need to know who to collaborate with, to understand it properly.

All the data you need may already exist within your Company, but finding all the relevant parts of it, in conjunction with the people to support understanding it, is hard.  In fact, it is so hard that you may not even try.  This results in re-creating existing knowledge, wasting time trying to get a task done without the information you needed, bias in outcomes towards the data that you did manage to find, a growing sense of frustration and – ultimately – increased risk to your business though bad or untimely decisions.

So, let’s take a look at some of the things that people try to do, to resolve these issues.

Single source of truth

The ideal is to have all your data in one consistent format, and many companies have undertaken huge projects to try and achieve this.  I don’t know for sure, but my educated guess is that all of these have failed – but not before huge amounts of time had been invested in building taxonomies and ontologies, and then training staff how to use the new search tools they will need.  Even then it’s not a full solution, because the resulting database only holds internal company data: additional external data is almost always needed to complete each project.

Ultimately, this ‘Data warehousing’ activity always results in failure – sometimes because employees become frustrated by not being able to use the tools with which they are familiar, and sometimes because the task of keeping the database up-to-date becomes overwhelming. This does not make for happy and productive workplace.


Joining of the sources to form a sudo point of truth with search

 The next most obvious solution is to use multiple sources of data and form some sort of layer across it all that supports a single search mechanism.  This sounds promising, but there are a few issues, from a technical standpoint.  Firstly, joining up many searches into one is not a simple task.  Also the search system becomes more unreliable each time a new source is added, because it is more complex to maintain and has more possible points of failure.  But a larger problem is that there is no way to compare the quality of the data that is returned from the different sources – so you the user still has to evaluate every separate search result, even though they performed a single search.  This is an unnecessary distraction which does not lead to a great experience, and thus reduces the chances of the system being used properly.

There is another way

The first step is to accept that many of the tools you employ are great, and that each team will be more efficient if they use the tools best for their job,.  Also accept that people will do the minimum they have to when storing information as it’s not really a core part of their skill set and it’s an annoyance. No matter how much training you provide they will always make errors, so don’t force them to categorise and add endless metadata to information with the aim of making it more discoverable in case it is needed in the future.  Ultimately the task of categorising data is just not scalable, since data and companies grow and change over time.

If you accept these things, you can have very happy groups of people throughout your organisation –  but you are still missing the key ingredient that builds the best culture and efficiency which is cross group collaboration, because each team is using its own data silo.  Bob in the core engineering team does not know that a document exists in the sales team, that means the wrong thing is being built, and Jon in business development does not understand why the services the company has are not in alignment with his goal.  No one understands how the work they are doing relates to the current company strategy (after all that is something that is discussed once a month at the regular company “all hands” and forgotten about 10 minutes later). 

Getting cross-functional groups of people to communicate is hard. If each group broadcasts everything they do, so that everyone else knows, then it’s just ignored – otherwise, everyone would have information overload. Occasionally someone will say ‘you should speak to Helen I am sure a while ago she mentioned something similar’ only to discover Helen left the company last month.  Alternatively, some proactive and extroverted employees go and talk to everyone they can think of to see if anyone can help, which wastes lots of time and is disruptive to everyone. Even if it does work, you will find that some teams are not forward enough to make these connections, so it is not a viable solution for the company as a whole.

In an ideal world, every employee would find the information they need, and make the connections with people they need, exactly when needed.  So, what if they could achieve that single point of truth without needing a single source of truth? This would support healthy engagement across the organisation and empower employees to make choices driven by company-owned information  in conjunction with external information, to support the best outcomes.  That is the goal, and there is a way you can get there without huge projects and investments in training …..

Achieving a Single Point of truth

What if you could leave all your data where it is, regardless of how unstructured it is, and still be able to discover information across all your company’s data sources, ranked in order of relevance?  Just discovering that a particular document exists will help understand who might be worth talking to, even it you can’t access it directly.  There is a way to achieve this with very almost no overheads for management and no overheads at all for employees, in the way they use the tools they are using.   iKVA have innovative solutions that achieve just this.  By indexing data using vector mapping technology all the data can be embedded into a single index, with links back to the original documents.  This index enables discovery of information across the organisation and beyond, and it can easily be embedded into existing tools to make the resulting insights available exactly when they are needed.  So all necessary information (both internal and external) is now at each employee’s fingertips, whatever tool they use and wherever the relevant data is stored.

Why wouldn’t you want to empower your workforce in this way?

Ian Firth is iKVA’s Chief Product Officer. He has 20+ years of product experience in the digital information sector. Former CPO of Speechmatics and Product Director at Citrix.

Like this article?

Share on Facebook
Share on Twitter
Share on Linkdin
Share on Pinterest

Leave a comment

This website uses cookies to ensure you get the best experience on our website.