Proving Alpha: How a Hedge Fund Measures AI's ROI - Hari Kumar, Contour Asset Management

0:01 [music] >> My name is Harry. Um, I am the head of AI at Contour Asset Management, which is a hedge fund based in the city. It's been around for a little over a decade

0:21 now, um, but I've been here for the past year-ish. Um, we're a fundamental long short shop, um, around like 15 20 people, 6 billion AUM.

0:32 Um, and mostly [snorts] invest in the TMT space. Um, so a lot of our investors are pretty tech forward by nature, but want to

0:41 bring a little bit more, um, AI workflow into the investment firm itself and see, okay, we've had all these processes done for many years in a very similar way. How can we kind of

0:53 shake things up and use AI to our advantage, especially when it comes to the research process, but then also bring that into trading, operations, compliance,

1:03 finance, pretty much all parts of the firm. So, came in in the past year and basically my mandate was, how can I show impact across all areas of the firm,

1:14 largely starting with this research process, right? And as probably all of you guys in this room can understand and appreciate, it's a really big challenge to show,

1:25 hey, we're spending this amount of money and time on AI spend and impact. What are we actually gaining from it? Both from a P&L standpoint, which is to come,

1:38 um, but also just from a more operational and are we getting good ideas perspective, too. The ROI capture from our investment is a

1:48 really big challenge to understand and capture, but I want to talk a little bit about some of the ways at in the investing landscape we're able to start to think about hey, we've put in this

2:00 amount of money into our AI spend, what are we getting out of it? Cool. So, a few things that we are thinking about

2:12 here and intentionally trying to keep this a little bit vague and more in terms of the operational and process parts of it, but down to dive into any technical stuff either at the end of the

2:21 session or afterwards as well. So, what we're trying to do is basically bring out live AB tests for some of the research workflows that our analysts

2:32 have. So, we have around 10 analysts that each cover different parts of our coverage, right? Which is how it works at a lot of these um hedge funds, especially on the smaller

2:41 side, where you have maybe one person that covers 30 to 40 names. And they've had their uh existing processes in terms of how they think about their investment

2:52 thesis, how they actually come up with new ideas, and then validate and test out over the long term whether those ideas actually came to fruition. But now in a AI and more agentic world

3:04 when it comes to investing, the challenge is would you have come to the same type of conclusion with the AI tools or without the AI tools as well? And a process that we've done in the

3:17 past few months that I think has been pretty successful has been actually creating live AB tests by recording and documenting their entire process that they would have done in a pre-AI world

3:30 and how they're doing it as we roll out more workflows and tools in the company itself. So, for example, what we did is for one specific name

3:40 when it came to our three-month cycle. So, from one earnings report to the following one, we tracked if they didn't use any AI tools. And this is at first just hey, if

3:52 you're not using any LLMs, what does your workflow look like? But then, as we've introduced more um tools at the hedge fund itself, like okay, you have some tools that help you

4:03 do expert call interviews better, or ones that help you field a consumer survey that's done by an agent instead of a human. What are your takeaways without things

4:13 like that? And then we're documenting, okay, you've done this process without AI, but also without these additional tech workflows that we're building out. What are the

4:23 data sources that you're finding? And what are the conclusions you're coming out with? Then we run that same process involving our PM

4:32 including some of these AI tools. And we're building them one by one with different things that we're coming at with, right? So, for example, when it comes to expert call interviews, one

4:41 thing that's been pretty helpful is you have expert calls that are run with an agent, and then you synthesize the notes yourself. How does that process work when you do

4:51 it with an agent work versus if you have the agent synthesize the notes for you, and then you come up with your conclusions at the very end of it, right?

5:00 So, it's kind of thinking about the question of how long would it have taken to source some of this information manually versus if you were able to do it in a

5:09 more agentic manner. This This is definitely a big challenge when it comes to the investing landscape because a lot of the work that we're doing is just based off of idea

5:20 generation. So, you're coming up with the new ideas and new theses, and you want to see, okay, can they be proven out in a short-term scale, which is a little less

5:30 of what we do, but at least in like the 3-month window, or are they things that take 6 months or a year to manifest in terms of an actual um impact to stock price, right?

5:42 So, the challenge with something like this is you can't show ROI immediately. It's more so from a process-based manner, right? So, are we improving the

5:53 process of how we're conducting consumer surveys? Are we improving the process of how we're updating our models? And one by one, we're trying to bring out a little bit more of um

6:06 uh ROI measurement when it comes to okay, are we actually changing the way someone's working or we just adding more thrust to their process? So, talked a little bit about these use

6:20 cases, but maybe just to touch a little bit more on them. The main goal when it comes to um fundamental investing is thinking of a thesis, but also as I've

6:31 learned, it's to not be so anchored to a thesis that you come up with initially and actually trusting some of the data, but also the expert and other insights that you're getting to

6:42 remove some of those anchors as well, right? And this is where I think that there's a lot of impact measurement when it comes to AI tooling that can be built. So,

6:52 example, um we have manual idea generation agents that are being run to take different ideas that are being come up with per name per analyst and adding more layers in terms of okay,

7:06 you've come up with a few different data points from different alternative data sets or you've talked to an expert in the industry to formulate some of that idea.

7:15 Let's layer on a few more conclusions or parts of the process to think about when it comes to how are you actually how are you actually understanding what

7:27 that idea is going to materialize in the next few months or if it's something that it takes even longer to take to take into account into the market. Additionally, we're

7:37 thinking a little bit more as to one of the biggest problems that we're seeing is you take an idea or you have an idea, you might work on it for a couple of days, and then that idea might

7:48 not be interesting to the PM or something else takes precedence, and you lose that idea. And it doesn't come back for months, and then maybe you miss like a key price window.

7:57 So, having agents that are also tracking through all of the notes that you've been taking, even if you only worked on an idea for a day, tracking it against the market and

8:07 trying to see, "Okay, is this now an idea that's interesting, and should we flash that as a point to not just you, but the PM and the trader as well?" And bringing out more quick work when it

8:18 comes to you've had a bunch of different things you've worked on, and maybe it's been weeks since you've looked at it. Now, let's get back to it because it's interesting again, right? I think that's

8:27 a big problem that a lot of hedge funds have had, where you have a bunch of different things that you can be working on, you have to prioritize, but the timing is actually the most important

8:37 thing as to when you want to strike when it comes to an interesting point in the market, right? So, building agents that are not only doing idea a generation, but thinking about timing and when you

8:48 should enter exit a position, even if it's not something that has been top of mind for you, is something that we've been really thinking about as well. Um and I guess the last point that I

8:58 want to make here, thinking a little bit more about the alternative data options when it comes to investing workflows, right? A lot of funds spend quite a good sum of

9:09 money when it comes to acquiring really new and novel data sets, right? And I think in this age, there's a bunch of different providers that are coming out with very nuanced solutions, even when

9:19 it comes to things like satellite mapping or really granular credit card data. Those things are helpful, but they're only helpful when they're all brought

9:28 together, right? So, if you think about something like a Yipit, which provides credit card panel data, that lives in isolation, but now with MTP and API access for things like Gipit, you can

9:40 feed it into our agents, which is kind of what we've done. So, it takes into account things like your expert call or notes transcripts, but also layers in different data sets when it comes to

9:51 credit card panels or spend data to see how those two things interact. Where before those have been differentiated, now we're trying to curb that issue as well.

10:05 Additionally, I think a challenge with measuring ROI is the time aspect of it. So, you spend a lot of money when it comes to acquiring some of this tooling when it comes to

10:18 research, model updates, AI spend. But, there's a lot of time when it comes to what is an analyst spending their day on, which is the main thing that we're thinking about, right?

10:29 So, what we've also done here, which I think has been pretty useful, is develop metrics when it comes to hey, these are the insights that have been generated by an analyst per day.

10:40 What was the time spent on that? So, tracking time spent on each name. And what were the eventual P&L outcomes from some from a decision like that?

10:52 That is something that is good in investing landscape because you have a very very obvious success metric when it comes to P&L. But, it takes a lot longer than other

11:02 industries to actually materialize, especially from a more fundamental shop, where you're not trading by the day, but you're making a few trades a day that are expected to materialize over

11:14 months-long horizons, right? So, understanding where analyst time goes is really valuable. And tracking things like a cost per useful insight, which is the metric that we've come up

11:25 with, really helps us understand, okay, you spent your time to generate these ideas, and those are all being tracked per day. Where do those ideas eventually land?

11:35 What were the interesting ones? And what were the tools that were taken to come up with an idea like that? So, that largely touches on the research side of workflows when it comes to

11:48 measuring um our ROI from not just AI spend, but general tech investment. But, we want to also improve this across the entire firm, right? It's

12:00 really important to make sure that we do quality research. That is how you get ahead. But, if you have better and more involved trading, as well as stronger ops and finance, that just helps the

12:11 firm at large to free up time to improve processes, right? So, for example, on the trading side of things, we're doing a lot more when it comes to measuring our entry and exit

12:21 times. Thinking about how our position is spread across multiple different industries and multiple names within an industry. And having AI agents track those names to make sure that our

12:32 traders are on top of it and understand, okay, these are interesting insights. So, bring them much more into the analyst world because their time should not just be spent on

12:43 executing orders. It should also be spent on bringing idea generation, feeding ideas to our analysts and associates as well. Additionally, on the op side,

12:54 this is something that I think a lot of um general companies can appreciate, but it's been pretty interesting to see it from a hedge fund landscape. Many of the

13:03 workflows that are being done are basically set within Bloomberg or set within your um order keeping agent. And those, while they've come up with some cool AI tools

13:15 to help improve the process, it's really challenging to integrate those together, which is something that we're starting to do here. So, how can you take all parts of the ops and

13:25 compliance workflow and marry them into one one specific tool that our ops and compliance um functions can use. So, they're not just

13:36 reliant on external providers to give them piecemeal solutions to hey, this is how you can do order reconciliation, or this is how you can uh track vendor info.

13:48 The goal that we're trying to create here is obviously minimize repetitive work. That's what everyone here is thinking about. But, specifically from financial operations, that is something

13:58 that can really be consolidated into one platform. Where right now you have to kind of go into many different platforms for different pieces of the order flow. So, bring those all

14:08 into one is um is a pretty big priority for us. And I think not necessarily to the point of this talk, a way to really capture ROI, but to understand how can you take

14:20 people who are working in trading and ops right now, and use their skills to improve the main goal of the fund, which should be to create create new ideas, understand the market, and focus on

14:32 timing. Okay. The the last piece that I want to talk about here is thinking a little bit about the existing spend that we have um when it comes to different data sources

14:45 that we're working with, and how we can kind of up-level that when it comes to hedge fund workflows. So, right now a lot of hedge funds will spend on

14:57 either they're spending on different piecemeal solutions like an AlphaSense for for data, a Yipit for alt data, or they're spending with um uh

15:08 holistic solutions that use um third party. So, something like buying a Roqoo membership, or investing in like Kensho, right? Those are really great solutions, and I think the the

15:19 goal of this eventually that we're trying to build in house, is how can you kind of bring all these tools that different analysts use at a hedge fund into one platform, but also up level the

15:32 spend that we have right now. So, for example, with an existing, let's say, AlphaSense subscription, which is a big um model and call plus data provider.

15:45 Before, a lot of analysts will just use the AlphaSense portal and pull from there and work within their existing models. Things like Rogo will kind of take that

15:55 one step further by allowing you to have all that data within your model itself, and you can have a platform that you chat with Rogo agents for. What we're trying to do is still use

16:06 things like an AlphaSense or Yipit tool, but bring them into our own platform internally that learns from our analysts analyst habits and is programmed specifically for each individual analyst

16:18 and their model pattern, so it's not all as one. And how can you increase the value of your subscription? A lot of these companies, obviously, are having like

16:27 MCPs and APIs that are being pulled out, but the goal should be to learn continuously from all of the different types of data and the model updates that we're getting from this as well.

16:40 And then, to wrap up here, I think the the last point of the ROI equation, too, has been a pretty interesting one on my end when it comes to, "Okay, you're bringing in new tooling to people who

16:52 are very tech forward for investors, but you understand that there are going to be places where this breaks down, where model calls break or the connector breaks down every couple days, and then

17:03 it's your job to fix it, right?" So, part of the ROI equation that we've been taking into account at Contour is trying to understand, "Okay, there are these pauses that we're

17:13 getting, but also what is the time that that being taken away trying to fix bugs, that someone from the tech side cannot fix itself, right? So, that's something that I think is pretty

17:22 important to consider when it comes to, "Hey, we're involving AI in investing workflows." It's all really exciting, and people are very into the concept of it, but we also need

17:32 to understand, okay, when it comes to time spent for, let's say, valuable insight, what is the time detracted to understand how these tools work as well. And that's something where that time is

17:42 decreasing as we go forward, but it's a metric that we're still considering. Um so, I think to wrap up, in investing workflows, measuring ROIs definitely a huge challenge for us, but

17:55 it's really important to be involved in this space because as AI adoption increases, especially at a lot of these smaller fundamental shops, having one unified layer that's doing a lot of

18:06 different tasks across analysts is going to be really, really valuable. Um I think that's all the time I have, but if anyone has any questions after the talk, would love to connect. Thank you.