Thought Leaders in Big Data: Interview with Sandy Steier, CEO of 1010data (Part 6)

Sramana Mitra: I think most of the big data solutions nowadays are being delivered in either public or private clouds.

Sandy Steier: But in that case, the purpose of the cloud is merely an outsourcing mechanism, so you don’t have to have your own computers and your own data center. It is not open to other people. If I put my database on Amazon EC2, that doesn’t mean you can get to it.

SM: What I am saying is the kind of business that you are enabling or your customers are interested in is to open that data up to third parties, their customers, and their partners. The cloud is an integral part of that service.

SS: That is correct. It is an interesting and powerful nexus between the cloud and big data. I would also like to add the following to the use case before: not only does Dollar General get good advice from Pepsi or Procter & Gamble, but the data is so valuable to Pepsi and Procter & Gamble that they are willing to pay for it. Dollar General, which has all this data, is actually now selling it. They are making more money with that business than they are paying for the data warehouse. Paying for an enterprise with a warehouse is an expense of tens of millions of dollars every year. We are much less expensive. But not only is it less, they are making money on it.

SM: I have a final question. What do you see as open problems from where you sit in this world?

SS: People are not thinking outside of the box enough. I think that if you look at what happens with things like Hadoop – by the way, we use very similar techniques to Hadoop, so I don’t have a problem with the technology per se – but Hadoop is a very difficult thing to use. You have to have a lot of knowledge to be able to use it. It seems like things have gotten harder, not easier. That is unfortunate. And that is because people are used to thinking in certain ways. They are used to “technology is technology,” and the “bigger and badder” technology is what you have to thrive for. Then people have to cope with it. So, you end up with a shortage of data scientists. What is a data scientist? It is a data analyst who also has to have a lot of technical knowledge to be able to deal with things like Hadoop and other big data technologies. There aren’t many people like that around. It is hard enough to find a qualified programmer or statistical analyst. To find them in the same person is very difficult.

I think – and this is very subjective – it should be much simpler. In our case, the paradigm we are using is the spreadsheet paradigm. We basically say data analysis should be as easy as using Excel. That was the going-in mission. If that is implemented more widely, or if similar things are implemented, then I think more people will be able to deal with large amounts of data more effectively. I think that people are thinking about it too much as technologists and not as business users.

SM: My bias is toward taking a problem and then figuring out what is the easiest technology you can use to solve the problem rather than starting a technology and then trying to find a problem to solve. I am based in the heart of Silicon Valley, and this is a disease here, to have technologies and to turn up with a solution looking for a problem.

SS: I didn’t want to call it a disease because I wanted to be more diplomatic, but I am glad you said it.

SM: This has been a very interesting interview, I really enjoyed it. Thank you very much for your time.

SS: Thank you.

This segment is part 6 in the series : Thought Leaders in Big Data: Interview with Sandy Steier, CEO of 1010data
1 2 3 4 5 6