Thought Leaders in Big Data: Interview with John Plavan, CEO of EarthRisk (Part 1)

John Plavan is the CEO, cofounder, and chariman of EarthRisk Technologies, a company that uses big data tools to create weather forecasts, especially in the fields of high and low temperatures. The special thing about EarthRisk systems is that they can create forecasts up to 30 or 40 days prior to any respective event, far more than traditional weather forecast systems. In this interview, John talks about the tools and systems EarthRisk uses to create its forecasts and how this affects the energy market. Further, he outlines potential opportunities for future entrepreneurs in this sector.

Sramana Mitra: John, let’s start with some context about you and EarthRisk and how that relates to the big data topic we will be discussing today.

John Plavan: The first thing I would like to say about big data – just to put in context my background and the company – I wouldn’t call EarthRisk a big data company per se. What I can say about EarthRisk is that we benefit from the advances in big data, computing power and the democratization of large-scale computing power. These developments have enabled us to start a company that probably wouldn’t have even been started without those advances in big data and data processing. I will get into the background of how we got started, but in general we use big data processes to do what we do rather than to say that we are a big data company.

SM: That is actually one of the interesting ways in which big data is being used. There are applications built on top of the big data technologies, and that is what is marketed.

JP: There is a large group of companies getting started because of what they can do now and couldn’t do before. Our company is one of those. If we have pretty aggressive theories we would like to test [to determine] if there are outcomes of the big data process that can be beneficial to somebody – those tests couldn’t be done before we had those tools available. Now that they are available, you can do a lot of what-if scenarios and see if you can find something that is useful. Once it is useful, everybody benefits. I think that is a great lead-in to how we got started here with our company.

As for me, I ran a variety of relatively small operating businesses. I had the good fortune to sell one of them that ran for about 16 years in 2008 and started a small venture capital/angel investor partnership. We were looking to commercialize technology that had been an output of research endeavors but that didn’t have a champion to take those technologies through research and figure out a commercial application for them. My partners and I are in San Diego, and we went down to some friends at the Scripps Institution of Oceanography, which is part of the University of California San Diego and renowned as a 100-year-old research institution. There were quite a few interesting technologies developed for specific research projects. Once those research projects were over, it is not necessarily in Scripps’ sight to try to find ways to re-purpose those technologies for a different application. Much like many other research institutions, Scripps just hadn’t done a lot of that.

While we were there, we came across a couple of interesting technologies and we ended up starting a couple of companies out of them. Once of them was EarthRisk Technologies. I have a business partner, Stephen Bennett, who is a lifelong energy meteorologist. When he was working on a big energy trading desk, as a meteorologist, trying to predict extreme temperature events in the future, he felt there had to be a better way. They spent a lot of laborious, time-consuming effort trying to compare what had happened in the past, what the pattern observation might be, and then use that to predict what might happen better than the standard weather models could do. It was a very laborious catalog comparison process, and he said, “There has to be a way to write some algorithms that can do this work and bring all of the millions of pieces of data of weather observations all over the globe, quantify them into specific weather patterns, and then compare those patterns with the resulting outcome that has happened in history.”

It is a huge data set and very difficult to do in a case-by-case process. He brought that idea to Scripps. The atmospheric scientists at Scripps did the research on it. They ended up finding that there are indeed quite a few relationships that you couldn’t really pick out without using a computer and trying to compare. Once those relationships were identified, you could use that to predict the increased risk of an extreme heat event or an extreme cold event, following the observation of a current set of patterns. Once that research was done, the relationships were identified and a 6,000-page catalog of data relationships was the output of the research.

At that point the scientists said, “We have proven the method. We have proven the fact that there is some new science here.” They are scientists. They didn’t necessarily want to create a software application that energy traders could use as a result of the output. So we formed EarthRisk Technologies to build software that will allow an end user to interface with the output of the big data processes that the atmospheric scientists developed.

When they had the idea of taking millions of pieces of information on weather observations all over the globe and using them as synaptic scale precursors for specific weather events, that just wasn’t possible a decade ago. Or at least it was not cheap, without using a large scale computing process. But now you can just contract computing instances in the cloud, write an algorithm to start the comparative processes, and a couple of days later you can see what you could find out. By enabling the scientists to test a variety of what-if theories, they came up with valuable information that allowed us to get a company started, and energy traders all over the world are now using our software to make decisions. It is a tremendous advancement and capability.

This segment is part 1 in the series : Thought Leaders in Big Data: Interview with John Plavan, CEO of EarthRisk
1 2 3 4 5