Skip to content

Investment Centre

How to measure asset manager skill

Identifying manager skill requires going beyond past performance. It involves a blend of quantitative analysis, statistical verification, and qualitative assessment. Statistical techniques such as hypothesis testing and regression analysis are crucial in trying to differentiate manager skill from luck. The above is a necessary but not a sufficient condition to assess future performance potential.
7 min read

Asset managers will almost always show their fund performance relative to some benchmark and any outperformance is often misunderstood to reflect the skill of the manager. In this article we endeavour to demonstrate that outperforming a benchmark does not necessarily mean that the manager is skilful, but rather that their performance could be the result of luck.

Importance of discerning skill

The difference between employing skilled and unskilled asset managers can result in a huge dispersion in returns. This dispersion widens as the investment horizon increases, compounding the importance of being able to identify skilful managers. For example, over the 12 months to 31 January 2024 in the ASISA High Equity Category, the fund with the highest return delivered 40.1%, while the fund with the lowest return lost 7.1%, a difference of 47.2% in a single year.

Outline of the challenge

It would be simple if all we had to do was pick past winners (best performing funds), implicitly assuming that these would represent skilful managers. However, the research (including work that we have done ourselves) suggests that this would not result in great outcomes. The disclaimer that past performance is not necessarily a guide to future performance is apropos. Teasing skill from past performance is notoriously difficult because capital markets are very ‘noisy’ (stochastic or random), which makes finding skill ‘a signal in the noise’ akin to finding a needle in a haystack.

Although past performance is far from useless, knowing how and when to use it requires a good understanding of the tools available to do this. More importantly, understanding the need for a qualitative approach and not just relying on quantitative assessments is critical.

The past versus the future

If that was not complicated enough, we also need to understand that most investors are interested in future performance, not past performance. The reason for trying to find skilful managers, is so that capital can be deployed with them, with the understanding that they are more likely to outperform in the future. Simply identifying skill in past performance would be of limited use. An example would be giving managers awards for their skilful past performance, although most awards do not even do that – they simply give awards to the best performing funds, not to the most skilful fund managers.

Quantitative analysis

  • Alpha generation: alpha represents a manager’s outperformance (or underperformance if alpha is negative) relative to an ‘appropriate’ (valid) benchmark. The word ‘appropriate’ does a lot of heavy lifting in that definition, disqualifying many benchmarks such as inflation and peer group averages. The problem, however, is that positive alpha alone, does not tell you whether a manager has skill (or more accurately, was skilful) or was simply lucky. This is often a major source of misunderstanding of alpha as many people do not recognise how to extract a signal from a random variable. In a later section, we will cover a statistical technique where we can do exactly this.

  • Risk-adjusted returns: sometimes alpha can be a simple product of the risk taken by the asset manager relative to the benchmark. Unfortunately, just calculating the alpha does not provide you with this important risk information. The industry has therefore designed risk-adjusted return metrics that adjust the alpha for the amount of risk taken to achieve it.

  • Benchmark comparison: as highlighted, using an ‘appropriate’ benchmark is critical in calculating alpha, but it is equally important in calculating active risk. This may be more obvious if you consider what active risk purports to measure, which is the dispersion of fund performance relative to the benchmark. If the benchmark is not appropriate, this relative performance is meaningless. A fund manager may have made the right call in investing in a specific stock, but not receive the appropriate credit for doing so if the benchmark does not recognise this. While we have already mentioned that some benchmarks may not be appropriate in general e.g. inflation, some may also not be appropriate for a specific fund as they do not represent how that manager is investing.

  • Consistency of performance: another way to measure performance is to simply calculate the proportion of outperformance of the benchmark using a certain frequency e.g. monthly or even daily. Here we ignore the amount of outperformance or underperformance, and simply focus on the relative frequency of each. If we assume that a manager simply picks stocks by flipping a fair coin, we would expect the manager to only get around 50% of their calls right, leading to outperformance. We need to recognise that this would be a random variable with a random outcome, and one could not simply infer that a number above 50% would be evidence of skill. Again, later we will describe how we can use statistical techniques to establish whether certain results suggest any evidence of skill or not. It is, however, important to note that you need to use non-overlapping periods when doing this analysis, and many people fail to recognise this, thus invalidating any results.

  • Attribution analysis: if you want to truly understand whether a manager is skilful or not, you need to understand where any outperformance has come from. This is where attribution analysis comes into play. If a performance measurement such as alpha, tells you how much the fund outperformed the benchmark, attribution analysis tells you what contributed to that outperformance.

    How the attribution is performed is important because it will only attribute the performance to the variables being measured. For example, if you wanted to understand a fund’s relative performance in terms of its asset allocation and stock selection decisions, you would need to use these variables in the attribution analysis. Again, simply assuming that a positive attribution value is evidence of skill fails to recognise that these are random variables requiring statistical analysis to separate any ‘signal’ from the ‘noise’.

Statistical analyses

  • Hypothesis testing: as mentioned above, simply using alpha or the proportion of outperformance as a measure of skill fails to recognise that these results are possibly random, and luck may be mistaken for skill. Fortunately, there are statistical techniques that help to put these metrics in the context of how random they are i.e. it allows us to extract any ‘signal’ (skill) from the ‘noise’ (randomness). Hypothesis testing is the specific tool that can be used to do this. Under certain assumptions, which should always be validated as they are often wrongly ignored, you can postulate that a manager has no skill under the null hypothesis, and then look for evidence that will allow you to reject that null hypothesis. This test can be performed on alpha and on the proportion of events – days or months – of outperformance.

    It is important to understand that these tests simply provide evidence in support of rejecting the null hypothesis that the manager does not have skill – or is failing to do this. It does not prove anything as statistics does not prove, but rather provides evidence in support of the null hypothesis, or the alternative hypothesis.

  • Regression analysis: another important tool in our toolkit is regression analysis, which helps to dissect manager performance in many different ways. In simple terms, a regression analysis helps to understand how independent variables may be used to explain fund performance. This can be very powerful in various exercises, including understanding a manager’s style bias – if they have one, or even that they do not have any – as well as the amount of systematic (beta) risk taken which was discussed above in relation to the Treynor ratio. Beta is the slope – sensitivity – of a linear regression of the fund’s performance relative to the benchmark, traditionally, a market index.

Qualitative analysis

  • Investment philosophy and process: unfortunately, quantitative analysis may not be enough to establish whether a manager has skill. We spend at least as much time on qualitative analysis as we do on quantitative analysis. The starting point in understanding a manager qualitatively, is to understand their investment philosophy and process. The investment philosophy is meant to reflect the manager’s alpha thesis – why they believe that they can outperform – and their investment process is meant to reflect how they organise themselves to harvest that outperformance (alpha). The investment process is extensive in range and scope, covering everything from how stocks are analysed, to how portfolios are constructed, including position sizes and sell disciplines. This is where you can start seeing how the future could be informed by the past. While many managers may not make revolutionary changes to their philosophy and process on a regular basis (unless something is seriously ‘broken’), many will keep evolving with the passage of time. Understanding how these changes may impact future performance is therefore critical in assessing whether a manager’s skill will persist.

  • Experience and track record: to think that the assessment of past performance is a purely quantitative and objective exercise, would be to misunderstand how much qualitative and subjectivity goes into analysing past performance. To begin with, analysing past performance has very little to do with simply comparing how a fund has performed relative to a benchmark or peer group. Instead, it includes many different dimensions of that past performance, including:

  • In addition, the team and philosophy and process together with other factors like fees, fund size, objectives (explicit and implicit) etc., are all critical in understanding past performance.

  • Team and resources: although last, people are in fact the most important qualitative factor that we consider. Almost nothing else matters if you have the ‘wrong’ people in the investment team. We need to evaluate people along various dimensions, including their qualifications, skills, and experience. This is done at the individual level, but the evaluation of the team, as a collective, is equally important. We are not believers in the ‘star’ portfolio manager philosophy, but rather in the wisdom of crowds. A diverse team of skilled individuals will, on average, outperform a single portfolio manager over time and will typically avoid outlier performance, both positive and negative. Although this may sound like it will lead to average performance, it is a poor understanding of how small positive results compound over time and reduce risk significantly. Finally, people do not operate in a vacuum – their access to data and systems/tools is very important. We therefore spend time understanding their resources to establish how these will support the investment philosophy, process, and people.

Conclusion

In the quest to identify manager skill, a multi-faceted approach is non-negotiable. Quantitative measures provide a starting point, but without the statistical methods to sieve through the randomness, as well as the qualitative assessment of the philosophy, process, and people, the analysis remains incomplete. It is the blend of numbers and narrative that brings us closer to the truth. A manager’s historical performance is a canvas, but the true art lies in interpreting the brushstrokes – the investment philosophy, the rigor of the process, the adaptability to market conditions, and the collective wisdom of the team. Evaluating a fund manager’s skill is as much about understanding the past as it is about anticipating the future. It is an exercise in balancing evidence with insight, and data with judgement. As the investment landscape evolves, so too must our methods of discerning skill, always cognisant of the complex, dynamic interplay between luck, talent, and the inexorable march of the markets.