Thought Leaders in Big Data: Interview with Josh Rogers, SVP of Data Integration Business at Syncsort (Part 2)

Sramana Mitra: From among the 2,000 customers you cater to, let’s do a few use cases of the kinds of business applications you are facilitating.

Josh Rogers: We see a few different scenarios where people are struggling. We have a number of customers that have made large investments in data warehousing environments. Perhaps they made investments in large-scale data integration platforms. What they are finding is that the ability to scale those architectures has become challenging. For example, they have a new information services company that wants to bring a new product to the market. To be able to do that, they need to combine the data sets of a few existing products as well as bring that data set on a refresh that combines those data sets on a weekly basis. To do that, they couldn’t scale their existing data integration platform to the volumes or the speed requirements needed to deliver their new product. So, they needed us to interact with their existing data integration platform to build scale up [along] the compute-intensive steps in that data flow.

What was specifically required was the ability to join five one-terabyte tables, provide an aggregation on that join, and serve that to a final data mart that would support the online experience. They would also need to be able to refresh that data weekly. What they found was that their existing data integration platform couldn’t handle the join or the aggregation in the time they needed. They use us to provide those two specific functions as part of the data flow.

SM: In terms of business problems you are solving, you described the technical challenges you are addressing. Can you shed some light on the business applications that involve these kinds of technical problems?

JR: I will give you two examples. One will be from the insurance business and the other will be from a new media business, which is around measuring Internet activity. We work with one of the largest property and casualty insurers in the nation. Their goal was to be able to understand the scope of relationships they have with their customers. If someone has a house and a car, the insurer would be able to make the next best offer on a weekly basis and push those offers out to their agents, so they could propose the best solutions to their customers. What they found was that they were unable to scale the aggregations required for the house-holding calculations to offer the next best offer because of the size of their customer base – they couldn’t do that aggregation process in the weekly turnaround they were looking for.

They used us to be able to increase that throughput and meet those weekly turnarounds. That is a very common scenario – there is a marketing offer or a customer insight that people want to look at on a weekly basis. That requires a huge set of customer and transaction data and being able to do very intensive roll-ups with that data.

From a new media perspective, there is an organization called comScore. Their mission in life is to measure Internet activity – to be able to understand who is visiting what property and for what reasons on the Internet. They want to understand not only what places people are visiting but also how they are getting there. They generate all sorts of detail reports for their customers so that people can place the right ads, reach the right demographic, and understand how customers are behaving with the information that is published online. To do this, comScore has two major data sets. One is a set of information that comes from opt-in panelists – people who opt in to have their website or the web surfing activity monitored by comScore.

This segment is part 2 in the series : Thought Leaders in Big Data: Interview with Josh Rogers, SVP of Data Integration Business at Syncsort
1 2 3 4 5