A guest post by Clif Triplett, ex-CIO, Baker-Hughes and member of Flutura advisory board.
“Big
Data”; another big idea like “Cloud”; is it something that we should be
embracing right now or be lost forever?
Is this another vague term that can be applied to countless
projects? Everyone is doing it our
vendors will tell us. You may have the question,
“Am I actually running a “Big Data” project or do I just have to deal with a
lot of data?” What is “Big”?
As I look forward, I think Big Data will
begin to take different distinct forms as this other concept of “Cloud” has
matured over recent years. Business
cases will become more obvious and solutions better defined. The enablers to address “Big Data” opportunities will also become more
obvious as we see the dynamics of tomorrow dramatically evolve compared to just
a few years ago.
The most influential contributors emerging
on the horizon that will change the game are the following:
- Explosion of sensors and
digital inputs
- Very large memory machines
(Terabytes of memory and more)
- Easy access to thousands of
processors from Cloud providers like Azure and Amazon
- HPC environments and Cloud
operating systems that can organize computing across many independent machines
- The continued explosion of
readily available data on the internet (e.g. weather, stock market, government
records, news, social media)
- Web crawlers that can locate
and index information on the web Predictive Analytic engines
- Open Source software (e.g.
Predictive Analytic engines, search engines, cloud operating systems, and I
anticipate the emergence of applications like those that might be able to judge
and predict sentiment about topics)
- Proven use cases by industry
- . Commercial-Off-The-Shelf (CoTS) package solutions connecting
multiple data sources and answering specific business challenges e.g. sales
forecasts, predictive maintenance, process control, inventory management.
This convergence of technology and
information now make it possible to define sub-classes of Big Data projects,
such as Demand Forecasting, Failure or Defect Management, Market Sentiment,
Situational Alternative Simulation, and many others I am sure.
Big Data product segments will also be clearer
and strong leaders will emerge. One of
the product segments where we will see specialization will be hardware
platforms. We will see specialized
machines such as HANA platforms and Solid State storage optimized for very
large data sets. We will also see Big Data Cloud platforms
leveraging the momentum of two major technology trends; Cloud and Big Data. There will be many analytic platforms; both
open source and proprietary. We will see
boutique solutions to specific problems like product launch predictive sales
analytics and advanced maintenance management programs.
The other big player to emerge will be
taxonomy platforms that assist in connecting disparate data sources and applying
context to support integration. Today
most major applications provide APIs to their applications. I foresee them in the future also providing
taxonomy context to allow more dynamic integration of systems.
So, what are some of the issues that this
future of Big Data will bring upon us in the near term? Organizational change management will be top
of the list. This emerging trend
surrounding Big Data will cause many new roles, new skill demands, new systems,
and new approaches to business. In short; change.
First of all, we will face the issue of
where is our Master Data and who is the owner?
This is a problem that has been with us for some time, but now it will
emerge as a very strategic decision and not just an IT annoyance. Second will be the issue of data quality and
how do we clean up all this conflicting or missing data we have uncovered. Our list will continue to grow and here are a
few more to consider:
- Development of Business Cases
to clean up data
- Engaging the business in data
governance
- New Technologies for solving
this new class of Big Data business solutions
- New skills required and a
shortage of top talent
- Technology and Business
roadmaps – what does it look like?
- Many new vendors emerging and
the old favorites trying new things they have never done before – how do we
know who knows what they are doing?
- Project estimating and run away
projects
We have seen many of these challenges
before and many of us will see them again.
It will be exciting times and we better start preparing now. The planets are aligning and Big Data is
arriving. It still may be a bit still too
far away for some to see it clearly, but keep your eyes open for it will be
upon you soon.