FIVE DATA CHALLENGES EVERY INDEX PROVIDER FACES

Partha Sen, CEO and co-founder, Fuzzy Logix

by Partha Sen, CEO and co-founder, Fuzzy Logix

As the popularity of index-based investing grows due to transparency and lower fees, pressure among index providers is increasing. And for good reason. Index-based funds that follow the S&P 500, Dow Jones and Russell 2000 for instance, tend to beat 80 percent of actively managed funds. This lower-fee style of investing makes a meaningful difference to an investor’s return. It should come as no surprise that last year, according to Morningstar passively managed funds attracted almost $505 billion, compared with outflows of $340 billion from active funds.

So what does this mean for index providers? The valuation and movement of an index can have an even greater impact on financial markets and investors. Index providers are under more pressure to meet the highest levels of data quality and precision in their calculations. But it turns out developing and managing an index is a complex and very time-consuming process that involves equal amounts of industry knowledge, mathematics and data analysis.

With all of the progress we’ve made in machine learning, algorithms and big data analytics, the industry has been surprisingly slow to adopt new technologies to streamline and accelerate the management of indexes and their time to market. Why? Historically it’s so operationally-intensive and the financial markets – and indexes – never take a vacation.

Processes across the industry are largely manual, and in some cases even date back 60-100 years. There are many different types of indexes, each with their own unique set of calculations. An index might be weighted, for example, based on a stock’s price or its market capitalization. Depending on the how an index is calculated, there are various factors to be considered such as the liquidity of constituent components, or the relative weightage of each component. Some key attributes and terms associated with an index include:

• Weighting Methodology – Indexes can be composed by weighting their constituents in different ways. Price-weighted, market-capitalization-weighted, and equally-weighted are three examples.
• Free-Float Adjusted Factor (FAF) – Not every share that has been issued in the market is freely tradable. Strategic holdings, for instance, are considered illiquid. The free-float adjusted factor is a measure of the ratio of liquid shares to the total.
• Cap Factor – In order to not allow anyone constituent of an index to have undue influence on its value, a cap factor may need to be employed to constrain its impact on the index.

In order to achieve the high degree of accuracy required, it can take several days and countless employee hours to manage an index. Processes include Index Review, Index Rebalancing, Index Back-testing, Ad-hoc Enquiries and Factsheets, Industry-sector Classification, Calculation of the index itself, after having worked out the FAF and Cap Factor. On any given day, challenges include:

1. Consolidating data from multiple internal and external sources (Reuters, Bloomberg, MarkIt, FactSet, Exchanges, etc.)
2. Ensuring data quality and eliminating data duplication
3. Analyzing large volumes of structured and non-structured data including securities trading data, market data, newswires, social media, annual reports, etc.
4. Extensive reviews and re-balancing of indexes
5. Back-testing and creating new indexes through a series of streamlined tasks such as universe selection, benchmarking against historical data and a maker-checker review process.

The key to managing a successful index today? The ability to sift through an enormous amount of data from multiple sources and to calculate it accurately and quickly. The good news is that progress is being made and some global market index providers are adopting new approaches.

Advanced data analytics and parallelized, machine learning algorithms that incorporate various index needs can automate processes and drastically alleviate bottlenecks to reduce employee hours and resources. This enables the index provider to automate many of its previously manually intensive functions – such as bringing rebalancing down to seconds instead of days – and eliminating any previous human error or data quality issues.

More importantly, by saving countless employee hours in operating a portfolio of indexes, modern day index management can help index providers shift resources to what will drive new revenue:  expansion and diversification of their portfolio by bringing new products to market quicker.

Partha Sen, CEO and co-founder, Fuzzy Logix

by Partha Sen, CEO and co-founder, Fuzzy Logix

As the popularity of index-based investing grows due to transparency and lower fees, pressure among index providers is increasing. And for good reason. Index-based funds that follow the S&P 500, Dow Jones and Russell 2000 for instance, tend to beat 80 percent of actively managed funds. This lower-fee style of investing makes a meaningful difference to an investor’s return. It should come as no surprise that last year, according to Morningstar passively managed funds attracted almost $505 billion, compared with outflows of $340 billion from active funds.

So what does this mean for index providers? The valuation and movement of an index can have an even greater impact on financial markets and investors. Index providers are under more pressure to meet the highest levels of data quality and precision in their calculations. But it turns out developing and managing an index is a complex and very time-consuming process that involves equal amounts of industry knowledge, mathematics and data analysis.

With all of the progress we’ve made in machine learning, algorithms and big data analytics, the industry has been surprisingly slow to adopt new technologies to streamline and accelerate the management of indexes and their time to market. Why? Historically it’s so operationally-intensive and the financial markets – and indexes – never take a vacation.

Processes across the industry are largely manual, and in some cases even date back 60-100 years. There are many different types of indexes, each with their own unique set of calculations. An index might be weighted, for example, based on a stock’s price or its market capitalization. Depending on the how an index is calculated, there are various factors to be considered such as the liquidity of constituent components, or the relative weightage of each component. Some key attributes and terms associated with an index include:

• Weighting Methodology – Indexes can be composed by weighting their constituents in different ways. Price-weighted, market-capitalization-weighted, and equally-weighted are three examples.
• Free-Float Adjusted Factor (FAF) – Not every share that has been issued in the market is freely tradable. Strategic holdings, for instance, are considered illiquid. The free-float adjusted factor is a measure of the ratio of liquid shares to the total.
• Cap Factor – In order to not allow anyone constituent of an index to have undue influence on its value, a cap factor may need to be employed to constrain its impact on the index.

In order to achieve the high degree of accuracy required, it can take several days and countless employee hours to manage an index. Processes include Index Review, Index Rebalancing, Index Back-testing, Ad-hoc Enquiries and Factsheets, Industry-sector Classification, Calculation of the index itself, after having worked out the FAF and Cap Factor. On any given day, challenges include:

1. Consolidating data from multiple internal and external sources (Reuters, Bloomberg, MarkIt, FactSet, Exchanges, etc.)
2. Ensuring data quality and eliminating data duplication
3. Analyzing large volumes of structured and non-structured data including securities trading data, market data, newswires, social media, annual reports, etc.
4. Extensive reviews and re-balancing of indexes
5. Back-testing and creating new indexes through a series of streamlined tasks such as universe selection, benchmarking against historical data and a maker-checker review process.

The key to managing a successful index today? The ability to sift through an enormous amount of data from multiple sources and to calculate it accurately and quickly. The good news is that progress is being made and some global market index providers are adopting new approaches.

Advanced data analytics and parallelized, machine learning algorithms that incorporate various index needs can automate processes and drastically alleviate bottlenecks to reduce employee hours and resources. This enables the index provider to automate many of its previously manually intensive functions – such as bringing rebalancing down to seconds instead of days – and eliminating any previous human error or data quality issues.

More importantly, by saving countless employee hours in operating a portfolio of indexes, modern day index management can help index providers shift resources to what will drive new revenue:  expansion and diversification of their portfolio by bringing new products to market quicker.