Making Sense of Unstructured Information

By David Hatch, Guest Author

Unstructured information is considered by many to represent over 80% of an organization’s data, and yet it remains largely untouched by conventional BI approaches such as query and reporting tools. This includes text analytics and federated search of customer service notes, call center dialog, web pages, web logs, word documents, emails, customer surveys and feedback, field service notes and other data not contained in a structured database or application table.

Organizations today capture volumes of unstructured data – words in the form of customer comments, survey or claim forms, service records, technician notes, e-mail messages, public records, and a variety of other documents. Corporate strategy is all about collecting information from many different sources, evaluating the probabilities of potential outcomes and then making intelligent decisions concerning the direction of the business. Yet many organizations have remained, for the most part, indifferent to their unstructured data sources that often reflect the true voice and sentiment of their customers, suppliers, and employees. In the past, little was done to use this data to better understand the business insight it contained because it fell outside of the well-established structured data filters.

Recent Aberdeen research shows that organizations are now starting to prioritize the use of unstructured data. Aberdeen Group conducted an in-depth research survey that has benchmarked companies’ current and planned direction toward integrating unstructured data with traditional structured information. The findings from Aberdeen’s July benchmark report – “Data Management 2.0, Making Sense of Unstructured Data” — show that the top 20 percent of companies that achieve increased performance (the Best-in-Class or BIC companies) have several characteristics in common. These companies have also found added value from previously untapped information sources. In fact, 24% of BIC companies rated the convergence of unstructured and structured information as one of their top priorities, and 56% of BIC ranked it as at least a high priority.

To remain competitive, the Best-in-Class are leveraging their rapidly growing unstructured and untapped data sources. The ability to transform unstructured data into relational data, and to open up information silos through enterprise search technology, enables a 360-degree view of everything pertinent to the business issue at hand. BIC companies are also more likely to have text analytics than the industry average and are a third more likely to have federated search technology in place. By incorporating solutions for both search and analysis of unstructured information, the BIC have improved employee productivity and customer insight, as well as reduced risk. In fact, Aberdeen’s research supports the improvement sited by BIC companies:

• 68% reported greater than 10 percent improvement in time spent searching for knowledge objects (employee productivity)
• 94% reported greater than 10 percent improvement in response time to customer demands
• 66% reported greater than 10 percent improvement in the ability to reduce risk by preventing harmful events before they occur
• 65% reported greater than 10 percent improvement in customer insight (“view of the customer”)

While e-mail, MSWord and MSExcel have been the initial sources from which the best-in-class have drawn unstructured data, the BIC are also aggressively investigating and planning future sources from which they will pull unstructured data. Web 2.0 sources such as Web logs (“blogs”) and wiki pages lead the way with 56% of BIC companies currently or planning to evaluate.

Information is usually considered a good thing. In fact, as a rule, more information leads to better business decisions. However, to be successful, organizations also need well defined strategies and support in place. Aberdeen’s survey results also showed that the organizations enjoying best-in-class performance shared several common characteristics with respect to their efforts to bring structure to unstructured data:

• 54% of best-in-class organizations have had a well defined data management strategy in place. The industry average follows with 31%.
• 48% of best-in-class organizations link their BI initiatives to their overall business strategy. The industry average follows with 31%.
• 42% of best-in-class organizations have a BI center of excellence in place. The industry average follow with 13%.

The term “Business Intelligence,” first coined in 1989, is going through a redesign. The intersection of BI, search, and text analytics will help support its new implied potential. The key lesson to appreciate is that valuable information may be found in nontraditional sources.