March 22, 2016

Why "Small" is good for Big Data ? 3 lessons learnt


                   Big data can be intoxicating or infuriating !

At Flutura, we have seen big data evoke 3 kinds of responses with customers specifically in the Industrial IOT area ( Energy sectors - Oil n Gas, Utility, engineering )
1. Intoxication - People get "high" on the possibilities.
2. Infuriated - People get mangled up and bury their heads in constraints.
3. An uncomfortable collision of the two
This article was born out of our real life experiences in grappling with the above range of emotions .Our response to this dilemma can be summarized in 6 words
Celebrate the small in the big
So what does this mean exactly ? Allow us to share 2 specific real life experiences from the trenches which shaped our mindsets
"Thinking Small" Experience-1 : THIN SLICE vs BOILING THE BIG DATA OCEAN;

How a small "thin sliced" data yielded big revenue potential for a leading Retail Energy Provider ?

We were working with a REP - Retail Energy Provider in the deregulated energy markets in Houston. REPs essentially intermediate energy distribution in the last mile between the large Utilities and the end customers in deregulated markets. This is a small margin business and REPs had to innovate to stay relevant to their customers thru value added services other than selling commoditized energy at a premium. As a part of the analysis it was discovered that about 1 % of customers guzzle 33 % of the energy - predominantly industrial and commercial customers. Smartly slicing the energy habits of this 1 % of customers unlocked monetizable blue ocean opportunities for the REP. This happened because they were able to surgically identify specific ways to reduce their energy consumption which unlocked millions of dollars in savings for the customer and energy audit/asset refinance opportunity for the REP. Truth be told - It did not need powerful big data infrastructure. It did not necessitate complex algorithms. A bunch of hungry folks asking deep questions was enough to bring tangible impact to the energy outcome.
The inclination to make things smaller + simpler really helped unlock a massive; revenue opportunity.

"Thinking Small" Experience-2 : ONE DEEP SIGNAL vs A PLETHORA OF 'NOISE' ATTRIBUTES

How a small innocuous signal - prosumption velocity, gave a new lens and galvanized a utility to alter consumer habits in Micro-grids ?

One of the trends impacting a Utility grid is the increasing movement towards distributed energy resources like solar and wind. This means the flow of energy is bidirectional. This also means that a consumer is a producer of energy too (A prosumer). A prosumers energy consumption and production habits are recorded thru smartmeters which record voltage,power and other energy events every 15 min intervals and this data is streamed into a massive NoSQL repository( Cassandra, Hbase) in real times. Fluturas energy data scientists zeroed in on one powerful signal to watch - Prosumption velocity - which quantifies the rate of change of bidirectional energy flow for all the consumers in the grid.This tells the story number of households and commercial/industrial customers becoming in a neighorhood who are shifting from being mere consumers of energy to producers of energy.Zeroing in on the vital few energy signals from the trivial many helped galvanize the micro grid command center into action. The insane focus on one specific signal stabilized the contribution habits of prosumers and transformed grid instability .
Therefore here are a few lessons learnt in "Thinking Small" for Big Data Projects in the Industrial IOT Big data world, which we would like to share with the larger community
  • Learning-1 : Size of business outcomes triumphs over size of input data. Consider saying “no” to data which can distract . Peripheral data takes your eyes off game changing use cases which can impact an important business outcome. Sometimes solving for one root problem and impacting the outcome can help business users perceive tangible value, build momentum and allocate budgets for big data projects whose business case needs to be hacked.
  • Learning-2 : Develop a reductionist mindset Celebrate Smallness - The best way to see big data is to see it small. At Flutura we believe that the disciplined pursuit of lesser data triumphs heroism associated with the scale of data processed. Here are a few questions we typically ask to hack our way to "small data"
  1. Is there One tiny metric which gives the perfect lens to explain the phenomenon and impact a disproportionate part of story ?
  2. Is there One small action to trigger which had massive outcomes
  3. Can I carve out a thin slice of data which had maximum revenue potential ? Where is the small data catchment which holds the most headroom for business impact ?
  4. How can we isolate the vital few signals from the trivial many ?
  • Learning-3 : "Sipping wine vs Gulping beer ": There is a lot of commonality between ingesting a drink and ingesting data. Some data pools that need to be savored like wine ( Visualize a White Zinfandel being sipped in small doses ) and some data pools need to be gulped like beer( Visualize a pint of Guinness going down fast :). Knowing when to treat your data like White Zinfandel and when to treat it like Guinness is key .
To conclude ,in today's day and age, 99 % of conversation is on the size of the input - petabytes /zeta-bytes of data,40.9 billion IOT devices emitting data etc, not on the size of the outcomes.

Can we flip the equation and celebrate the size of outcome over size of data ?

Happy to hear your thoughts :)

1 comment:


  1. Nice information about Hadoop.
    The best place to learn hadoop is steinmetzils
    100% Job assurence is provided by them.
    visit: http://www.steinmetzils.com/

    ReplyDelete