From Big Data to Intelligent Applications

Is there a bigger purpose for all the work that small and large companies are doing in the Big Data domain? Bigger than trying to find needles of insight in haystacks of data?

My answer is a definite _Yes_. Big Data’s ultimate purpose is to make humans more intelligent! It’s to save us from some (yes, not all!) of the dumb decisions that we make!!

Doing things more intelligently by enhancing our natural capabilities via some sort of tool has been a dream not unlike the dream of building the iron horse or the flying machine. These are the type of dreams that were/are immensely challenging for humans to bring to reality but we just cannot give up on achieving them. Building these tools are part of our innate desires to explore the world beyond what is naturally available to us.

With our ability to record and analyze large amounts data from different sources in realtime, we are making one more leap towards bringing the dream of enhancing our intelligence closer to reality via human-made tools. Using the flying-machine analogy, we are now able to build balloons that get us off the ground in terms of enhancing our intelligence (for example, easy access to all information on the Web). Similar to how understanding the physics principles that led to building the flying machine, there has been good progress in understanding the principles of how intelligence works in the human brain. This will hopefully lead to building the equivalent of 'intelligence flying machine’ but we still have a lot of hard work to do to achieve that.

Going back to the reality of today’s state of practice in building tools that enhance our intelligence in significant ways, I think one obstacle is the divide between the Big Data side of things and the Intelligent Applications side. I propose a lifecycle view of what I call Big Data to Intelligent Applications. It’s a generalization of the projects that I have either worked on directly or seen from other people’s work that have leveraged the recent progress in tools and approaches in analyzing Big Data and creating applications that amplify and enhance the intelligence of their users in making better decisions or taking the right actions.

The lifecycle starts from the data sources that can vary widely in their sources and formats. Next, there is the stage that is perhaps still the hardest to do (i.e., requires manual work and is harder to automate), and at the same time receives the least attention and credit! This is the stage that data from disparate sources and with different format has to be prepared in a way that can be added to the semantic graph. Semantic graph is a data artifact (a composite artifact) that makes data amenable for machine reasoning. Then there are a set of ‘computing engines’ that operate on the semantic graph to enable machine reasoning. The outputs of machine reasoning are then translated into a set of capabilities in the applications that in turn enhance or amplify the capabilities of the human users to predict events, discover insights, detect anomalies, etc.

The Big Data to Intelligent Applications lifecycle view depicted here may seem trivial or obvious to some of the experts in the field. To some, it may look too generic to be useful as a blueprint for building intelligent applications. To others, there might be places where the lifecycle is too specific (for example the semantic graph as the only way to encode intelligence into the data). But I hope for people who are focused on either side of the lifecycle, namely Data vs. Intelligence, it helps in bridging the two sides.