Thought Leaders in Big Data: Interview with Josh Rogers, SVP of Data Integration Business at Syncsort (Part 4)

Sramana Mitra: So you are basically accelerating the sort process.

Josh Rogers: I would say we started with that basic technology on the mainframe. What we are doing today is accelerating all sorts of correlated processing functions. Whether that is aggregation, a join or a filtering process, we have technologies that allow you to accomplish that core architectural benefit of making that workflow go faster but at the same time using less CPU.

SM: Given what you are doing and where you sit in the ecosystem – you have several partnerships in the system integration domain – what are on your radar as interesting open problems? Where is this sector of the industry going? Where are the trends and the opportunities?

JR: This is a good time for me to talk about the world of big data, how we define that and specifically how new architectures like Hadoop are creating significant opportunities for customers but also a series of new challenges to take advantage of. From a big data perspective, our core assumptions are the following: One is that it is going to be a requirement for organizations of all sizes and across most industries to be able to take advantage from insights from big data in order to remain competitive.

We believe that big data is going to break companies’ architecture. We don’t believe that companies’ traditional approaches to solve their data integration challenges or even their analytics challenges are going to hold up under the scale, variety and velocity of big data. We believe the customers are going to have to look for new architectures to be able to solve these problems and gain the insights they are looking for to be competitive in their industries. Hadoop is clearly one of many architectures we see being adopted. More so than some others we believe Hadoop is emerging as the key operating system for big data. There are lots of different ways of looking at the adoption of this technology vs. the adoption of others, but what you will find is that it is outstripping most other core enterprise IT architectures that have been adopted over the last 20 years.

What is going to drive the adoption of Hadoop is the ability to lever key killer applications on top of this Hadoop architecture. Today most of these applications are being custom coded through Java, but you are going to see more and more vendors starting to release applications that allow customers to get value quickly out of their Hadoop cluster. Specifically we believe that ETL is the leading use case today, and probably the killer app that will drive the adoption of Hadoo and give people the ability to take the use cases I described before and deliver those in a Hadoop architecture that is going to allow them to deliver a lot more data to their analytics environments and ultimately their business users at a fraction of the cost and at a scale that had not been envisioned five or ten years ago, when they built the traditional data warehousing architectures. At Syncsort we have looked at the Hadoop architecture and we have come up with a couple of premises.

The first one is that the Hadoop architecture is fundamentally a better architecture to support core data integration requirements because it [protects] the data by allowing you to have cheap storage of the data, store data in any different format you might need and because of the Map Reduce framework in the way that it works and executes omnidata – it gives you the massive ability to scale up core data processing functions. But we have also noticed that the Map Reduce framework, while it gives you great scalability because you can scale out across commodity hardware, is actually not that performing, specifically in the sort step. We believe there is an opportunity not just to improve the performance of that environment, but also by delivering an easier way for people to use it for large-scale ETL. We believe that will drive massive adoption.

This segment is part 4 in the series : Thought Leaders in Big Data: Interview with Josh Rogers, SVP of Data Integration Business at Syncsort
1 2 3 4 5