The Silent Rockstar of BigData: Machine Learning

Sure, world is crying out loud that big-data’s biggest problem will be resources. Demand has skyrocketed and everyone in the world is going into tailspin in meeting that demands. Companies are going frantic and overspending to hire data scientists to secure themselves from any upcoming shortfall. This is nothing but a sign that world needs our robot algorithm friends to pacify some demand and increase credibility to new paradigms. Who could forget Steve Balmer’s famous quote comparing Big Data as a Machine Learning problem. For starters, consider following 5 pointers why Machine Learning will have the last laugh when it comes to handling big-data first hand.

Too much data and too few people: Firstly, this is a no surprise that machine learning algorithms will work at the pace not matching their counter scientist friends. If trained properly, machine could easily pacify majority of data preparation and analysis demand in data analytics world. Another cool thing about machine learning is that once code is prepped and machine is programmed, you could use it multiple times and multiple places and see the magic happen. The trick is to not overkill first but to use it for overhead tasks first and keep making it more and more sophisticated, so that it will start doing all the heavy lifting and pacifying the resource demand as a result. Hence, machine learning single handedly can reduce big-data resource crunch and make the resource distribution relevant and appropriately.

Continuous Data Discovery: Now another place that is easy to tease is the insane ability of machine learning to keep digging and go into a continuous foray of data discovery. You could use the engine to it’s limit. It can scale beautifully as you increase the cluster and resources with it. So, if you have a data discovery module or algorithm, you could use it to any extreme to make it continuously surf for data driven discoveries for your organization and keep delivering the value as long as lights and data is continuously provided. So, if you care to build a R&D facility that does not work from 8am to 5pm, machine learning could really come handy helping your organization. So, the more data driven your organization is, the better machine learning prospects will be for your data discovery hunt.

Taking heavy lifting away: If you have been dealing with data you must have heard that 80 percent of data analytics is data prep. So, imagine seeing your resources manually working through large data sets combing the filters to prepare data for further analysis. That is one job that almost every data scientist hates and cannot skip. So, that is a good area for machine learning algorithm to chime in and they are best at it. You could train algorithms to find patterns between data and bring them in clusters. This will do the majority of heavy lifting so that data scientists could spend their time on area that is really bang for the buck.

Unknown Unknowns finder: So who could forget the famous quote:
“There are known knowns; there are things we know that we know.
There are known unknowns; that is to say, there are things that we now know we don't know.
But there are also unknown unknowns – there are things we do not know we don't know.”
—United States Secretary of Defense, Donald Rumsfeld.

Not sure if I agree to most of other things he said, but this is a line that resonates best with current big-data ecosystems and machine learning is providing a huge service to make sure unknown-unkowns are found and some story could be build on their creations spreading more light on their existence. Machine learning in it’s adaptive and learning full mode could help figure out those unknown-unkowns and surface them to known-unknowns. The more it plays the better we get at surfacing unknowns. This is what machine learning brings to the arsenal that other data scientists could have difficulty in.

Possibilities are endless: So, now we get that machine learning could work endlessly, they scale well, they could do heavy lifting and they help surface unknown issues. Now another thing that we could benefit from machine learning is by bending their rules and using them at innovative places to surface hidden insights. So, unlike humans where industry expertise is of supreme importance, machine-learning algorithms could easily be trained to work in totally different domains. Like any other technologies out there, machine learning enjoys the same benefit of being industry agnostic and easily adaptable to new markets and opportunities. So, possibilities are endless and algorithms could be twisted and turned and tested on new opportunities, which makes them special as well.

So, machine learning could really take big-data and reduce human dependence from it and as data volume, velocity and variety is increased machine learning will have a better chance to cope up as Moore's law makes their system faster, cheaper and smaller. So, we should keep our hopes up and radar on in adopting machine learning in our corporate DNA. So, we could cope up with changing world with increasing complexities and scale.

Now, a disclaimer, I am not trying to say that machine learning will replace the data scientists. I am trying to make a case that computers are best at certain things and they will keep getting better, faster and cheaper, so it makes sense to use them in areas where they could take most of the burden away from data scientists. So, they need to be treated as friends and not competitions to create a win-win scenario.

Originally posted on http://analyticsweek.com/the-silent-rockstar-of-bigdata-machine-learning/

To view or add a comment, sign in

Insights from the community

Explore topics