October 16, 2013

Upstream Sensor Data + Big Data Analytics = Game Changer in Oil n Gas industry

Each blow costs the Oil and Gas industry a Billion dollars.

Can we avoid it? Can we see it coming and take action? 

We also know that each operating rig consists of thousands of sensors. This sensor data is used to analyse and reduce HSE (Health, Safety and Environment) risk considerably and dollars.

To show in detail how this is done, Flutura will explore breakthroughs achieved with upstream oil and gas data pools and Big Data analytics. This would transform the way HSE (Health, Safety and Environment) risk is detected by unlocking patterns previously not seen. Let’s begin, by asking ourselves some key leading questions that may help us diagnose the problem better and help transform the way we have been dealing with this problem so far:

1.       How can disruptive innovations in big data processing help mitigate blow out risks in the Oil and Gas industry to save lives and secure our environment?

2.       What are the key data blind spots in data upstream in the rig?

3.       What are some of the powerful unanswered questions from MWD (Measurement while drilling) / LWD (Logging while drilling) data?

4.       How can we triangulate across data streams to see key early warning signs upfront?

5.       What if we can wear new lenses to see risk patterns previously undetected?

The Oil n Gas industry is ripe for innovative solutions as two factors are converging to create the perfect storm.

·         Disruption enabler-1 : An audacious problem related to Health, Safety and Environment ( HSE) issues

·         Disruption enabler-2 : Large headroom to juice untapped data assets in upstream part of Oil n Gas value chain

Flutura calls this next generation approach - Oil n Gas 'Last Mile' Safety Risk Analytics 2.0. This approach hits at the intersection of Sensor data + Big Data Platform Engineering + Deep Machine Learning. This is a definite turning point from the traditional ways of looking at data using Historian and/or traditional database systems for assessing risk.

Most risks which happen in the Oil n Gas industry are emanating from events which happen in the last mile. If one is able to triangulate on sensor and human generated event streams to detect early warning signs which have not been seen, then one can detect signals early on. Flutura believes that the best way to mitigate risk is by reacting strongly to weak signals in the last mile and this involves continuous triangulation  of  streaming MWD/LWD and SCADA data streams over a long period of time using a combination of platforms + people
So what does the last mile in the Oil industry look like? The last mile in the oil industry is a sensor jungle which consists of a variety of MWD (Measurement while drilling) and LWD (Logging while drilling) sensors and SCADA devices. Thousands of these sensors are measuring multiple things from drill rpm, mud flow rates, CO2 gas emissions, valve positions, pump states and in the process emit billions of events. By using innovations in Big data like Map reduce and horizontal scaling, the Oil and Gas  industry has enormous potential for  unlocking previously unseen patterns buried in the data which help reduce risk in the last mile.

1.       Is there a co-relation between the out of bound of drill rpm rhythms from MWD Sensor data and operators experience?

2.       What are the frequent sequence of events (mud flow, Drill pressure, temperature, CO2) exhibited in rhythm disturbances prior to a near miss event which can trigger shutdown actions before an adverse event occurs?

3.       Which parameters where out of preset bounds (2 sigma events) before an incident happened - pressure / temperature or rpm so that the checklists on recalibration can be fine tuned?

4.       Are there keyword frequencies in maintenance inspection notes like "vibration", "leakage" which can be mined from text mining process which gives early warning signals to an impending incident?

5.       Where their signals in identity management logs which provide a clue to forensically investigate safety incidents?

6.       What is the NBI- "Next Best Intervention" for a Drill, valve, pump etc based on its historical behaviour?

The data to answer the above questions is available in the last mile sensor data captured as illustrated below:

Throwing data-blind-spots into the spotlight

There are devices like pumps, valves and sensors attached to a drill which emit either change of value events or alarm events. These events can be stored in a central event repository instantiated on a Hadoop cluster with massively parallel processing capabilities which can scale in an economic manner. Storm can be used for event stream processing, PIG to do computations in data pipeline and Hadoop /HIVE to do batch based computation

3 key best practices to mitigate risk in Oil n Gas (HSE) using big data

So what are additional best practices which can be leveraged to mitigate HSE risks?

1.       TRIANGULATE :
Look for patterns which reinforce each other across SCADA/MWD/LWD/Surveillance video event streams - structured, unstructured, internal and external.

- What are the top 3 cross event stream triangulations which are important signals to watch out for?

Consider having organisational processes which can pick the weakest link and amplification processes for field intervention. An alert prioritisation framework which rank orders signals would help risk specialists respond in a surgical fashion

- How can a weak signal be picked up and transmitted in real time to a point of action?

Be conscious of last mile data blind spots be it contractor data, incident notes or device logs.

- How can we build a last mile integrated 360 degree integrated data picture?
We at Flutura believe that answering powerful unanswered questions and executing the 3 key best practices would dramatically heighten situational awareness and mitigate operational risks.

New machine learning and big data innovations in processing billions of events and a variety of messy data sources using Map reduce paradigms has helped a lot of human generated companies to disrupt their markets. The question is how can Oil and Gas companies use the same data innovations levers on machine data to disrupt the way of managing last minute risks?

As Marcel Proust said “The real voyage of discovery consists not in seeking new landscapes, but in having new eyes”, we at Flutura too believe that discovering HSE (Health, Safety and Environment) risk consists not in looking for more systems but in seeing with ‘new analytical eyes’ to look at previously collected data.

Srikanth Muralidhara
Flutura Decision Sciences & Analytics


  1. The expansion of internet and other business intelligence leads to large volume of data. Industries are looking for talented professionals to maintain and process huge volume of data with latest tools available in the market.
    Big Data Training in Chennai|Big Data Training

  2. تعد شركة ركن الامانة (0505561372) من الشركات المتخصصة فى اعمال التنظيف و النقل و المكافحة بالمنطقة الجنوبية فى المملكة العربية السعودية فهى تقدم خدمات متميزة وعلى اعلى مستوى من الخبرة و التقنية بالاعتماد على عمال مدربين ومتميزين فى عمالهم ونقدم خدمات باقل الاسعار التى تناسب جميع عملائنا الكرام فتواصلوا معنا على رقم الشركة
    شركة تنظيف بخميس مشيط
    شركة تنظيف منازل بخميس مشيط
    شركة تنظيف كنب بخميس مشيط
    شركة تنظيف خزانات بخميس مشيط
    شركة عزل اسطح بخميس مشيط
    شركة عزل خزانات بخميس مشيط
    شركة كشف تسربات المياه بخميس مشيط
    شركة تسليك مجارى بخميس مشيط
    شركة مكافحة حشرات بخميس مشيط
    شركة نقل اثاث بخميس مشيط

  3. Oil & Gas industry loses billions of USD yearly just because of Non-Productive Time and an Industrial Internet of Things (IIoT) solution can help the industry out. An Industry that spends trillion of USD yearly on development, exploration, and asset maintenance, even a single-digit percentage improvement by implementation of IIoT will result in massive cost reduction.

  4. Your new valuable key points imply much a person like me and extremely more to my office workers. With thanks; from every one of us.
    offshore safety course in chennai

  5. I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.
    safety course in chennai

  6. Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.
    fire and safety course in chennai

  7. Great post! I am actually getting ready to across this information, It’s very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.
    Data science training in pune
    Data Science Interview questions and answers
    Data science training in bangalore

  8. This comment has been removed by the author.

  9. A skilled Data Scientists study issue, imagine designs crosswise over information, investigate information and utilize their insight and experience and finding to pick procedures and parameters that might be pertinent to the careful issue close by. data science course in pune

  10. I was blown out after viewing the article which you have shared over here. So I just wanted to express my opinion on Data Analytics, as this is best trending medium to promote or to circulate the updates, happenings, knowledge sharing.. Aspirants & professionals are keeping a close eye on Data analytics course in Mumbai to equip it as their primary skill.


  11. Attend The Digital Marketing Courses in Bangalore From ExcelR. Practical Digital Marketing Courses in Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Digital Marketing Courses in Bangalore.
    Digital Marketing training in Bangalore

  12. Attend The Data Analytics Courses From ExcelR. Practical Data Analytics Courses Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analytics Courses.
    ExcelR Data Analytics Courses

  13. Such a very useful article. I have learn some new information.thanks for sharing.
    data scientist course in mumbai

  14. Well, The information which you posted here is very helpful & it is very useful for the needy like me.., Wonderful information you posted here. Thank you so much for helping me out to find the Data Analytics Course in Mumbai Organisations and introducing reputed stalwarts in the industry dealing with data analyzing & assorting it in a structured and precise manner. Keep up the good work. Looking forward to view more from you.


  15. Excelr is providing emerging & trending technology training, such as for data science, Machine learning, Artificial Intelligence, AWS, Tableau, Digital Marketing. Excelr is standing as a leader in providing quality training on top demanding technologies in 2019. Excelr`s versatile training is making a huge difference all across the globe. Enable ​business analytics​ skills in you, and the trainers who were delivering training on these are industry stalwarts. Get certification on " data science in hyderabad" and get trained with Excelr.

  16. Thanks for sharing your valuable information to us, it is very useful.
    data science course

  17. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.
    data scientist certification