Four Ways to Overcome Data and Analytic Challenges in the Insurance Industry

By John Morrell, Sr. Director of Product Marketing, Datameer

The insurance industry, in particular the property and casualty, life and annuity, and re-insurance sectors, is fraught with very interesting data and analytics challenges. While there is vast potential for big data and advanced analytics such as Artificial Intelligence (AI) and Machine Learning (ML), data challenges often hold insurance carriers back from fully exploiting this potential.

I chaired a panel at an insurance event earlier this year and the event was very revealing into the data and analytics issues of the insurance industry.

One key takeaway was that while the insurance industry does not produce physical assets, insurance carriers should be in the business of producing one key asset: Data. Insurers have tremendous volumes of data about customers, risk, losses, policies, claims and more that can be transformed into real business value.

Like manufacturers who optimize their business around the production of their physical assets, the key to optimizing an insurer’s business is to streamline the production and use of data. Similar to how high performing organizations embrace digital transformation and use data assets to execute more effectively in the market, insurers that effectively produce and utilize data assets will be the higher performers in their sector.

The insurance industry faces a number of unique challenges due to regulation, individually operating lines of business, varying depths of data sources, and more. Modern digital age competitors are challenging the business models of incumbents, making them embrace digital transformation or become extinct.

One of those major challenges is putting in place a broader data strategy to make more business-ready information available to analysts and business teams. A 2016 study by Willis Tower Watson, surveyed insurance industry executives about their use of big data. One question asked probed into the challenges and barriers the executives see. Second, behind people skills, is data availability (see figure 1).

Figure 1: Barriers and Challenges to Big Data Analytics

Let’s explore four ways insurance companies can expand their data strategies to drive forward thinking and unique analytics that can propel them to become high performing organizations and leaders in their markets.

1. Break Down Those Silos

Many insurance carriers, like other industries, have multiple data silos dispersed across their organization. In the insurance world, this happens due to independently operating product teams or lines of business, which focus on their own data needs and performance. In some cases, these lines of business came via acquisition.

Data owners, typically the line of business, place barriers to sharing data across the business through slow, cumbersome processes that are IT resource heavy. Lack of resources and complex processes are to blame for carriers creating an effective enterprise-wide data strategy.

Carriers can focus on creating a modern data management architecture that can break down data silos and create an aggregated view of various aspects of the business. With the proper data pipeline platform:

  • IT and data teams become enablers and leaders, facilitating the proper sharing and use of data
  • Centralized or business-embedded analytic teams can create their own data pipelines, unique to their needs, to drive faster insights and explore new questions

To quote one of my panelists, “I don’t know what I don’t know”. And expanded access and use of data helps explore the unknown.

2. Aggregate to Get 360 Degree Views

Data silos present a data architecture challenge, regardless of industry. In the insurance world, they prevent carriers from getting complete views on customers, risk, losses, product performance and more.

A 2016 study by Willis Tower Watson, surveyed insurance industry executives about their potential use of big data. In this study, executives revealed where they were using big data today – 2016 – and where they expected to use big data two years later – 2018 (see figure 2).

Figure 2: Use Cases for Big Data Now and Two Years Later

Not surprisingly, the top areas were getting better overall views of:

  • Pricing, underwriting & risk
  • Potential losses and claims
  • Understanding customers

Getting to the heart of these issues requires an agile enterprise data strategy and data pipeline platform that can aggregate the data from various siloes and facilitate faster delivery of business-ready information to analytics teams. This allows business analysts to get 360 degree views of customers, loss potential and more to drive next best actions that increase overall customer lifetime value, and make informed decisions to assess total risk exposure.

Banking firms have recognized the need to get broader perspectives on areas such as customers and risk, the former driven by competition and the later driven by regulatory compliance. In the end, they turned to agile data pipeline platforms to facilitate faster aggregation and organization of data.

3.      Don’t Strive for Perfection

In many ways, the final presentation at the CDAO Insurance event was the most revealing. One very interesting point by the speaker, Gene Boomer, Senior Enterprise Data Architect from Protective Insurance, was that insurers should not strive for perfection, but rather achieve the “most.” This meant:

  • Delivering data in the most timely manner
  • Gather the most complete data
  • Creating the most trust in the data

In this case of insurance carriers, and quite frankly any industry, more is always better. Creating the perfect data architecture and data delivery processes is, in practical terms, unachievable.

This means carriers should strive to focus on enabling broader data pipelines and more access to data, one step at a time. Start with high value use cases, focusing on the data needed to deliver those. Expand from there to reach other areas of the business. Banks often started in risk or customer-facing processes, then drove adoption in additional lines of business.

The greater the adoption and use of data via these pipelines, the more trust will be built in the data that is being delivered. After all, trust is a function of how analytic peers view the usefulness of the data. Trust will be managed socially via self-built data pipelines that analysts can properly share.

4. Govern with an Even Hand, not a Stick

In a regulated industry such as insurance, governance will play a vital role in an effective data architecture. Often times it is issues such as data privacy and governance that allow owners of data silos to put up barriers to data sharing.

Governance, however, needs to be managed with an even keel and with an eye towards driving broader use of data. I’ve encountered some organizations so fixated on data privacy and security that it paralyzes the use of data assets to drive business transformation and execution.

Yes, data security and privacy is important, but this this can effectively be balanced with data sharing. A more appropriate and positive way to describe how you govern data is permissible access. Provide access to those who can properly use the data in effective business situations, and don’t take an attitude of blocking first.

As insurance carriers enter the early stages of a data strategy, it is important to facilitate use cases where deep governance is less of a requirement and will not be a barrier to adoption. Broader governance policies can be defined and implemented over time as greater data democratization is facilitated.

It is also important to recognize that governance is 50% process and 50% features in the data pipeline platform that enable proper governance. Governance processes are often unique to each business and organization, and therefore need to be tailored to those specific needs.


Creating a broader data architecture could be viewed as a large project that would involve a large number of tools that need to be stitched together. In the early stages of a data strategy, however, it is critical to minimize technology risk. This requires focus on the near term use cases, but keeping an eye toward future needs.

A data strategy with a a data pipeline platform as a data hub enables an organization to mitigate risk in the early stages, and evolve data delivery pipelines and processes over time to meet the broader need of the organization. It allows a strategy to stay focused, create repeatable processes and evolve policies such as governance over time as adoption grows.

Are you ready to create and grow your data strategy? Do you want explore how a data pipeline platform can facilitate the creation and use of data assets with the proper blend of governance? Learn more at