Machine learning is now part of many conversations about equity research. For a portfolio manager or chief investment officer, the more useful question is rarely whether a model is clever. It is whether the team can understand what the model is doing well enough to rely on it, challenge it and explain it to someone else.
That is what explainability means in practice. This article sets out what it can and cannot offer an investment team, and what to ask of any system that claims to provide it.
Two different questions
When people ask a model to explain itself, they usually mean one of two things.
The first is a global question: in general, which characteristics does this model pay most attention to? For an equity model, the answer might be that earnings quality, balance sheet strength and valuation carry the most weight across the universe as a whole.
The second is a local question: why did this particular company rank where it did today? Here the answer is specific. Company A might rank highly because its profits convert reliably into cash, while its share price volatility holds it back.
Both matter. Global explanations help a team judge whether a model's logic is consistent with its investment philosophy. Local explanations help an analyst decide whether a specific signal deserves their time.
What a useful explanation looks like
A good explanation for an investment team tends to share a few qualities.
It uses factors the team recognises. An explanation in terms of familiar concepts, such as earnings quality, leverage or valuation, is easier to test against experience than one expressed in abstract model features.
It shows direction as well as size. Knowing that a factor mattered is less useful than knowing whether it pushed a ranking up or down.
It is recorded. An explanation that exists only on screen at the moment of decision is hard to review later.
It sits alongside the inputs and the model version. Without these, it is difficult to reconstruct why the same company was treated differently six months apart.
What explainability cannot do
It is worth being clear about the limits.
An explanation describes how a model reached an output. It does not prove that the output is right. A model can be wrong for reasons its explanation describes perfectly well, for example because the data it relied on was incomplete or because a relationship that held in the past has stopped holding.
Explanation methods are also approximations. Some techniques estimate the contribution of each factor rather than reading it directly from the model, and different methods can give somewhat different answers. That does not make them useless, but it does mean an explanation should be treated as evidence to examine, not a verdict.
Finally, explainability does not remove the need for judgement. A clear explanation of a poor decision is still a poor decision. The value lies in making it easier for experienced people to spot when something looks wrong.
Why it matters for oversight
Investment teams answer to more than their own standards. Clients, boards, compliance functions and, where relevant, regulators increasingly expect firms to be able to show how decisions were reached.
A model whose outputs cannot be explained creates an awkward gap. The team may find its signals useful, but it struggles to describe its process with confidence. Explainability helps close that gap by making the model's role visible: what it proposed, why, and what the people responsible did with that proposal.
This is also why explainability works best when it is designed in from the start rather than added afterwards. If forecasting and allocation are kept as separate steps, for instance, each can be explained and examined on its own. When everything happens inside one opaque process, it becomes much harder to tell which part was responsible for an outcome.
Questions to ask of any system
If you are assessing software that claims to be explainable, these questions can help.
Are explanations given for individual decisions, for the model overall, or both?
Are the factors expressed in terms your team would use?
How are explanations calculated, and what are the known limitations of that method?
Are explanations stored with the inputs, model version and any human review?
Can you see how explanations change over time for the same company?
What happens when the model and your team disagree, and is that recorded?
A practical standard
For an investment team, explainable AI is less about the sophistication of the technique and more about a practical standard: could a colleague who was not in the room understand why a decision was proposed, and what was done about it?
If the answer is yes, the model is supporting the investment process. If not, it may be adding a layer of complexity that is difficult to oversee.
This article is educational and general in nature. It is not investment advice and does not describe the performance or validation status of any product. Intelligent Wealth's software is under development.