October 16, 2013

Upstream Sensor Data + Big Data Analytics = Game Changer in Oil n Gas industry

Each blow costs the Oil and Gas industry a Billion dollars.

Can we avoid it? Can we see it coming and take action? 

We also know that each operating rig consists of thousands of sensors. This sensor data is used to analyse and reduce HSE (Health, Safety and Environment) risk considerably and dollars.

To show in detail how this is done, Flutura will explore breakthroughs achieved with upstream oil and gas data pools and Big Data analytics. This would transform the way HSE (Health, Safety and Environment) risk is detected by unlocking patterns previously not seen. Let’s begin, by asking ourselves some key leading questions that may help us diagnose the problem better and help transform the way we have been dealing with this problem so far:

1.       How can disruptive innovations in big data processing help mitigate blow out risks in the Oil and Gas industry to save lives and secure our environment?

2.       What are the key data blind spots in data upstream in the rig?

3.       What are some of the powerful unanswered questions from MWD (Measurement while drilling) / LWD (Logging while drilling) data?

4.       How can we triangulate across data streams to see key early warning signs upfront?

5.       What if we can wear new lenses to see risk patterns previously undetected?

The Oil n Gas industry is ripe for innovative solutions as two factors are converging to create the perfect storm.

·         Disruption enabler-1 : An audacious problem related to Health, Safety and Environment ( HSE) issues

·         Disruption enabler-2 : Large headroom to juice untapped data assets in upstream part of Oil n Gas value chain

Flutura calls this next generation approach - Oil n Gas 'Last Mile' Safety Risk Analytics 2.0. This approach hits at the intersection of Sensor data + Big Data Platform Engineering + Deep Machine Learning. This is a definite turning point from the traditional ways of looking at data using Historian and/or traditional database systems for assessing risk.

Most risks which happen in the Oil n Gas industry are emanating from events which happen in the last mile. If one is able to triangulate on sensor and human generated event streams to detect early warning signs which have not been seen, then one can detect signals early on. Flutura believes that the best way to mitigate risk is by reacting strongly to weak signals in the last mile and this involves continuous triangulation  of  streaming MWD/LWD and SCADA data streams over a long period of time using a combination of platforms + people
So what does the last mile in the Oil industry look like? The last mile in the oil industry is a sensor jungle which consists of a variety of MWD (Measurement while drilling) and LWD (Logging while drilling) sensors and SCADA devices. Thousands of these sensors are measuring multiple things from drill rpm, mud flow rates, CO2 gas emissions, valve positions, pump states and in the process emit billions of events. By using innovations in Big data like Map reduce and horizontal scaling, the Oil and Gas  industry has enormous potential for  unlocking previously unseen patterns buried in the data which help reduce risk in the last mile.

1.       Is there a co-relation between the out of bound of drill rpm rhythms from MWD Sensor data and operators experience?

2.       What are the frequent sequence of events (mud flow, Drill pressure, temperature, CO2) exhibited in rhythm disturbances prior to a near miss event which can trigger shutdown actions before an adverse event occurs?

3.       Which parameters where out of preset bounds (2 sigma events) before an incident happened - pressure / temperature or rpm so that the checklists on recalibration can be fine tuned?

4.       Are there keyword frequencies in maintenance inspection notes like "vibration", "leakage" which can be mined from text mining process which gives early warning signals to an impending incident?

5.       Where their signals in identity management logs which provide a clue to forensically investigate safety incidents?

6.       What is the NBI- "Next Best Intervention" for a Drill, valve, pump etc based on its historical behaviour?

The data to answer the above questions is available in the last mile sensor data captured as illustrated below:

Throwing data-blind-spots into the spotlight

There are devices like pumps, valves and sensors attached to a drill which emit either change of value events or alarm events. These events can be stored in a central event repository instantiated on a Hadoop cluster with massively parallel processing capabilities which can scale in an economic manner. Storm can be used for event stream processing, PIG to do computations in data pipeline and Hadoop /HIVE to do batch based computation

3 key best practices to mitigate risk in Oil n Gas (HSE) using big data

So what are additional best practices which can be leveraged to mitigate HSE risks?

1.       TRIANGULATE :
Look for patterns which reinforce each other across SCADA/MWD/LWD/Surveillance video event streams - structured, unstructured, internal and external.

- What are the top 3 cross event stream triangulations which are important signals to watch out for?

Consider having organisational processes which can pick the weakest link and amplification processes for field intervention. An alert prioritisation framework which rank orders signals would help risk specialists respond in a surgical fashion

- How can a weak signal be picked up and transmitted in real time to a point of action?

Be conscious of last mile data blind spots be it contractor data, incident notes or device logs.

- How can we build a last mile integrated 360 degree integrated data picture?
We at Flutura believe that answering powerful unanswered questions and executing the 3 key best practices would dramatically heighten situational awareness and mitigate operational risks.

New machine learning and big data innovations in processing billions of events and a variety of messy data sources using Map reduce paradigms has helped a lot of human generated companies to disrupt their markets. The question is how can Oil and Gas companies use the same data innovations levers on machine data to disrupt the way of managing last minute risks?

As Marcel Proust said “The real voyage of discovery consists not in seeking new landscapes, but in having new eyes”, we at Flutura too believe that discovering HSE (Health, Safety and Environment) risk consists not in looking for more systems but in seeing with ‘new analytical eyes’ to look at previously collected data.

Srikanth Muralidhara
Flutura Decision Sciences & Analytics


  1. The expansion of internet and other business intelligence leads to large volume of data. Industries are looking for talented professionals to maintain and process huge volume of data with latest tools available in the market.
    Big Data Training in Chennai|Big Data Training

    1. Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. IEEE Projects for CSE in Big Data But it’s not the amount of data that’s important. Final Year Project Centers in Chennai It’s what organizations do with the data that matters. Big data can be analyzed for insights that lead to better decisions and strategic business moves.

      Spring Framework has already made serious inroads as an integrated technology stack for building user-facing applications. Corporate TRaining Spring Framework the authors explore the idea of using Java in Big Data platforms.
      Specifically, Spring Framework provides various tasks are geared around preparing data for further analysis and visualization. Spring Training in Chennai

  2. تعد شركة ركن الامانة (0505561372) من الشركات المتخصصة فى اعمال التنظيف و النقل و المكافحة بالمنطقة الجنوبية فى المملكة العربية السعودية فهى تقدم خدمات متميزة وعلى اعلى مستوى من الخبرة و التقنية بالاعتماد على عمال مدربين ومتميزين فى عمالهم ونقدم خدمات باقل الاسعار التى تناسب جميع عملائنا الكرام فتواصلوا معنا على رقم الشركة
    شركة تنظيف بخميس مشيط
    شركة تنظيف منازل بخميس مشيط
    شركة تنظيف كنب بخميس مشيط
    شركة تنظيف خزانات بخميس مشيط
    شركة عزل اسطح بخميس مشيط
    شركة عزل خزانات بخميس مشيط
    شركة كشف تسربات المياه بخميس مشيط
    شركة تسليك مجارى بخميس مشيط
    شركة مكافحة حشرات بخميس مشيط
    شركة نقل اثاث بخميس مشيط

  3. Oil & Gas industry loses billions of USD yearly just because of Non-Productive Time and an Industrial Internet of Things (IIoT) solution can help the industry out. An Industry that spends trillion of USD yearly on development, exploration, and asset maintenance, even a single-digit percentage improvement by implementation of IIoT will result in massive cost reduction.

  4. Your new valuable key points imply much a person like me and extremely more to my office workers. With thanks; from every one of us.
    offshore safety course in chennai

  5. I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.
    safety course in chennai

  6. Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.
    fire and safety course in chennai

  7. Great post! I am actually getting ready to across this information, It’s very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.
    Data science training in pune
    Data Science Interview questions and answers
    Data science training in bangalore

  8. This comment has been removed by the author.

  9. A skilled Data Scientists study issue, imagine designs crosswise over information, investigate information and utilize their insight and experience and finding to pick procedures and parameters that might be pertinent to the careful issue close by. data science course in pune

  10. I was blown out after viewing the article which you have shared over here. So I just wanted to express my opinion on Data Analytics, as this is best trending medium to promote or to circulate the updates, happenings, knowledge sharing.. Aspirants & professionals are keeping a close eye on Data analytics course in Mumbai to equip it as their primary skill.


  11. Attend The Digital Marketing Courses in Bangalore From ExcelR. Practical Digital Marketing Courses in Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Digital Marketing Courses in Bangalore.
    Digital Marketing training in Bangalore

  12. Attend The Data Analytics Courses From ExcelR. Practical Data Analytics Courses Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analytics Courses.
    ExcelR Data Analytics Courses

  13. Such a very useful article. I have learn some new information.thanks for sharing.
    data scientist course in mumbai

  14. Well, The information which you posted here is very helpful & it is very useful for the needy like me.., Wonderful information you posted here. Thank you so much for helping me out to find the Data Analytics Course in Mumbai Organisations and introducing reputed stalwarts in the industry dealing with data analyzing & assorting it in a structured and precise manner. Keep up the good work. Looking forward to view more from you.


  15. Excelr is providing emerging & trending technology training, such as for data science, Machine learning, Artificial Intelligence, AWS, Tableau, Digital Marketing. Excelr is standing as a leader in providing quality training on top demanding technologies in 2019. Excelr`s versatile training is making a huge difference all across the globe. Enable ​business analytics​ skills in you, and the trainers who were delivering training on these are industry stalwarts. Get certification on " data science in hyderabad" and get trained with Excelr.

  16. Thanks for sharing your valuable information to us, it is very useful.
    data science course

  17. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.
    data scientist certification

  18. Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon.
    ExcelR data analytics

  19. I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post. Hats off to you! The information that you have provided is very helpful.
    ExcelR Data Analytics courses

  20. I am looking for and I love to post a comment that "The content of your post is awesome" Great work!
    data science courses in bangalore with placement

  21. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!
    ExcelR Business Analytics Course

  22. Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing.
    ExcelR Business Analytics Course

  23. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!
    ExcelR data analytics courses

  24. Its as if you had a great grasp on the subject matter, but you forgot to include your readers. Perhaps you should think about this from more than one angle.
    Please check this ExcelR data science course pune

  25. Attend The Data Analytics Course From ExcelR. Practical Data Analytics Course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analytics Course.
    ExcelR Data Analytics Course

  26. Awesome..I read this post so nice and very imformative information...thanks for sharing
    Click here for data science course


  27. The blog is very informative.Keep posting like this.
    Aviation academy in chennai

  28. good article about data science has given it is very nice thank you for sharing.
    Data Science Training in Hyderabad

  29. Thank you for excellent article.You made an article that is interesting.
    AWS Solutions Architect courses in Bangalore with certifications.

  30. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more. data science course

  31. Wow what a Great Information about World Day its exceptionally pleasant educational post. a debt of gratitude is in order for the post.Please check ExcelR Data Science Courses

  32. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.
    ExcelR Data Analytics Course
    Data Science Interview Questions


  33. I have been searching to find a comfort or effective procedure to complete this process and I think this is the most suitable way to do it effectively. ExcelR Data Science Class In Pune

  34. It’s always so sweet and also full of a lot of fun for me personally and my office colleagues to search your blog a minimum of thrice in a week to see the new guidance you have got.
    Angularjs Training in Chennai
    Java Training in Chennai
    Bigdata Hadoop Training in Chennai
    SAS Training in Chennai
    Python Training in Chennai
    Software Testing Training in Chennai

  35. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.
    data analytics course mumbai
    data science interview questions

  36. Great post i must say and thanks for the information.However, is still among the leading topics of our time. I appreciate your post and look forward to more.
    Data Science courses
    data analytics course
    business analytic course

  37. Actually I read it yesterday but I had some thoughts about it and today I wanted to read it again because it is very well written.
    Please check this Data Analytics Course in Pune

  38. keep up the good work. this is an Ossam post. This is to helpful, i have read here all post. i am impressed. thank you. this is our data analytics course in mumbai
    data analytics course in mumbai | https://www.excelr.com/data-analytics-certification-training-course-in-mumbai

  39. I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!

    Orthodontist in Bangalore

  40. I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!

    machine learning course

    artificial intelligence course in mumbai

  41. This is a wonderful article, Given so much info in it, Thanks for sharing. CodeGnan offers courses in new technologies and makes sure students understand the flow of work from each and every perspective in a Real-Time environmen python training in vijayawada. , data scince training in vijayawada . , java training in vijayawada. ,

  42. First You got a great blog .I will be interested in more similar topics. i see you got really very useful topics, i will be always checking your blog thanks.. Please check this Data Scientist Course

  43. One stop solution for getting dedicated and transparent Digital Marketing services and We take care of your brands entire digital presence.
    The digital marketing services we provide includes SEO, SEM, SMM, online reputation management, local SEO, content marketing, e-mail marketing, conversion rate optimization, website development, pay per click etc. We will definitely promote your brand, product and services at highest position with consistency in order to generate more revenue for your business.Digital Marketing Services & Trainings

  44. Nice blog. I finally found great post here Very interesting to read this article and very pleased to find this site. Great work!
    Master's in Data Science Programs in Houston

  45. Nice blog,I understood the topic very clearly,And want to study more like this.
    Data Scientist Course


  46. I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.
    data science course

  47. It's late finding this act. At least, it's a thing to be familiar with that there are such events exist. I agree with your Blog and I will be back to inspect it more in the future so please keep up your act.. Please check this Data Scientist Course

  48. Attend The Machine Learning course Bangalore From ExcelR. Practical Machine Learning course Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Machine Learning course Bangalore.
    Machine Learning course Bangalore

  49. The information provided on the site is informative. Looking forward more such blogs. Thanks for sharing .
    Artificial Inteligence course in Jaipur
    AI Course in Jaipur

  50. You might comment on the order system of the blog. You should chat it's splendid. Your blog audit would swell up your visitors. I was very pleased to find this site.I wanted to thank you for this great read!!
    Know more Data Science Courses