The Portfolio Manager Stress-Testing AI's Financial Reasoning

By Mark Holloway, CFA — 2026-02-24

Mark Holloway has spent fifteen years watching financial models fail in ways their builders didn't anticipate. As a portfolio manager at a leading asset management firm and later a research lead at a major macro hedge fund, he developed a specific skill: finding the edge cases where a quantitative model behaves well in normal conditions and badly in the ones that matter.

He brought that skill to IXO two years ago. The work felt familiar. A new category of model — large language models applied to financial reasoning — was entering the market with the same pattern he'd seen before. Impressive average performance. Systematic blind spots. Failure modes invisible without domain expertise.

The practitioner gap

"Financial AI tools were being evaluated by people who understood machine learning but not markets," he says. "You need both. A model can produce a coherent-sounding analysis of a portfolio's rate sensitivity and still be wrong in ways that would be obvious to anyone who's actually managed duration risk through a rate cycle."

One project involved evaluating an AI system designed to assist institutional analysts with credit analysis. The model performed well on standard cases. Holloway's job was to find where it didn't.

Finding the invisible errors

"I gave it a scenario involving a leveraged buyout with unusual debt covenants and a sector-specific liquidity dynamic. The model produced a clean, well-structured analysis. It also missed the thing that would have made a senior credit analyst pause immediately — a covenant interaction that created refinancing risk under a specific rate scenario."

The annotation he provided didn't just flag the error. It walked through the reasoning chain: why the covenant interaction mattered, what historical precedent it echoed, what a practitioner would have done next.

Intelligence vs. judgment

"The gap isn't intelligence. These models are genuinely intelligent about finance in the aggregate. The gap is judgment — knowing when the aggregate pattern doesn't apply. That's what fifteen years of managing actual money gives you. You've seen the cases where the model would have been wrong. You remember them."

Read The Portfolio Manager Stress-Testing AI's Financial Reasoning on the IXO blog