The following appeared in Logic Magazine and was posted on Medium.com 

In recent years, we’ve been hearing more about the role of algorithms in our lives. Algorithms are helping decide whether people get a job or a loan, what news (fake or otherwise) they consume, even the length of their prison sentence.

But algorithms are also rewiring the world of finance, with immensely important consequences. In an era of financialized capitalism, finance plays a powerful role in the global organization of wealth and work — which means that everyone will feel the effects of the industry’s transformation.

We sat down with an algorithmic trader to learn more about how algorithms are remaking the industry, and why it matters. We talked about what algorithmic finance actually looks like, who the winners and losers are likely to be in the new big data gold rush, and why we may be entering an era of irrational cyborg exuberance.

Let’s start by talking about your background. How did you get involved in finance?

I was always interested in economics and had a quantitative background. Anyone who succeeds academically where I grew up ends up being very quantitatively oriented. After school, as I was trying to find a profession that would be financially rewarding but would also allow me to use what I studied, I started looking at the financial industry. I ended up taking a job on a trading floor in an investment bank.

Most large banks have at least one, typically several trading floors. It’s an actual floor, about the size of a football field, filled with traders who do business with large investors looking to trade stocks, bonds, or futures, or to borrow money. The bank makes money by taking a commission, or by “market-making” — intermediating between buyers and sellers, taking some risk with its own money while it waits for the two sides to match up.

When I think of a trading floor, I think of a bunch of guys screaming into the phone, ‘Wolf of Wall Street”-style.

It’s not so much people yelling into the phones anymore. The trading floor has evolved quite a bit over time. It used to be more about being alive to the transactional flow of global markets. It’s increasingly about the operations that enable that flow, and the intellectual property that allows people to make money off that flow.

I liked it. The trading floor is still where a lot of the actual design and transactions of global markets take place. And it’s stimulating. If you want to use your intellectual muscles, you can do so pretty quickly. You’re not just sitting at a desk somewhere out of the way, or trying to pitch corporate titans with some arbitrary analysis to back you up — which can be more of a salesmanship game and less of an intellectual exercise.

Anyway, over time I migrated to the investment strategy part of the financial world. I started helping large asset owners — entities like pension funds and sovereign wealth funds — allocate their money to systematic investment programs. That’s where I migrated to because that’s where most of the financial world was migrating to after the 2008 financial crisis, as everyone realized that the old ways of investing were not really doing what they wanted them to do.

Portfolios had been too exposed to the same underlying risks. Technology was now enabling investors to understand their risks better, and to take more direct control over their investments. Part of the shift involved removing human decision-making when it wasn’t perceived as adding any value.

What do those new ways of investing look like? What are “systematic investment programs,” and how do they fit into the field of algorithmic finance as a whole?

There are many ways that algorithms are actually used in finance, so the term algorithmic finance gets used more loosely than it should.

There are at least two major domains in which algorithms dominate. The first is what’s frequently called algorithmic trading, which focuses on market microstructures. It programs computers to make split-second automated decisions on how stocks are bought and sold. Should you buy a whole bunch of shares at once? Or should you split up your purchases over time? How do you more intelligently execute trades? Algorithmic trading uses algorithms to help answer these questions — and it’s an enormous industry. There are a lot of hedge funds and traditional investment banks that try to make money there.

The other domain, which is the one that I’m more focused on, is sometimes called systematic investing and sometimes called quantitative investing. It’s also very much algorithmic investing. It involves using algorithms to allocate money systematically based on data.

An early version of quantitative investing — starting roughly in the 1950s, with the birth of Modern Portfolio Theory — was designed to create rules to save for retirement. These rules were supposed to help people decide how much of their money to put into stocks and how much of it to put into bonds. Once you’ve made that decision, you have a rule that lets you allocate money across stocks and bonds at some defined frequency automatically, without a human being going in and having to make any qualitative decision. This basic framework was rapidly adopted across investment portfolios at every scale, from mutual funds for individual investors to asset allocation decisions by the largest funds in the world.

Then people took that framework and applied it to an increasing number of underlying assets, with a much finer degree of granularity. So now you’re not just making rules that determine the overall mix of stocks and bonds in a portfolio but which stocks, which bonds, which commodities, which corn futures, and so on. And the rules used to distribute assets become far more complex.

When you say “making rules,” what exactly are we talking about here?

These rules didn’t need any further human intervention, in the sense that they completely defined a portfolio. But in practice, investment wasn’t completely rule-based: investors used the model outputs as a baseline, and then tweaked it with their own decisions. Human insight could further improve the asset mix, in a variety of ways. Investors might want to buy cheaper stocks, for instance, or “time” the market by getting in or out at the right time. Or they might want to find new assets like commodities and mortgage securities, or improve the measurement of risk.

More recently, however, advances in computing power and financial engineering have vastly expanded the universe of analytical tools that can be applied to investing. The latest “rules” involve developing machine learning models that train on large amounts of data. It could be data from the financial statements of publicly traded companies. It could be macroeconomic data. It could be the price history of certain financial instruments. It could also be more esoteric data like satellite imagery.

What’s a concrete example of an investment decision driven by a machine learning model?

You could purchase a “sentiment” score developed by a firm that trawls Twitter on a continuous basis to understand changes in the mood of the market, or around a specific company. You could use that data to train your model, which could then determine whether to buy or sell certain shares. Usually signals like a sentiment score decay pretty quickly though, so you would want to be able to make that trade fast.

How automated would that process be? Are we talking about software making recommendations to human traders, or actually executing trades itself?

The level of human oversight varies. Among sophisticated quantitative investors, the process is fairly automatic. The models are being researched and refined almost constantly, but you would rarely intervene in the trading decisions of a live model. A number of hedge funds, mutual funds, and exchange-traded funds (ETFs) run on auto-pilot.

By contrast, most traditional investors use models to provide guidance rather than to generate automated trading decisions, since it’s unlikely that they could operationalize a complex trading strategy.

One of the challenges with machine learning is explainability. As the model becomes more complex, it can become harder, even impossible, to explain the results that it generates. This has become a source of concern as public scrutiny of the tech industry has increased, because you have algorithms making decisions that affect people’s lives in all sorts of ways while the reasoning for those decisions remains completely opaque.

When the financial industry plugs a bunch of data into a model in order to make an investment decision, how important is the explainability of the result?

I think the result should be very explainable. But that’s not a universal view. In fact, there’s a fairly big split between people who have concluded that explainability is holding back the advancement of the use of these techniques, and the people who hold on to the rather quaint notion that explainability is important.

But to some extent, explainability was already an issue well before we started using machine learning, because even traditional models of investing were hampered by some of these same issues. Finance is not like physics. You have a lot of feedback loop mechanisms impacting how participants interact with financial markets.

To give you a simple example, you might look at the price data of a stock and conclude that because that stock went up last month, it’s a good idea to buy that stock today. And if you do that systematically, you might expect to make some money. But if everybody else comes to the same conclusion, then the stock could get overbought today based on the movement of the stock over the past month. And if it’s overbought, you might actually expect to lose money on it over the next month.

Looking at historical data to figure out where your investment is going to go is useless if you haven’t thought about the mechanism by which it’s going to do that. In the example I gave, if you didn’t have an explanation for why the stock was moving the way it was moving, you might have missed the fact that the underlying mechanism didn’t really exist, or that it wasn’t robust enough to weather a whole lot of market participants looking to take advantage of that phenomenon.

So explainability has been an issue for a while. Everyone is always looking for a story for why they’re doing what they’re doing. And many of those stories aren’t that robust.

But isn’t there a strong financial incentive to try to understand why you’re doing what you’re doing, whether it’s an algorithm or a human executing the trades? Otherwise it seems very easy to lose a lot of money.

Sure. But the market structure of investing dilutes that incentive.

The people who are developing the most sophisticated quantitative techniques work for hedge funds and investment banks. For them, there are two ways to make money. You make money by charging fees on the assets you manage, and you make money on the performance of the fund. That split will give you a sense of why there’s a dilution of the incentive. Because even if your assets don’t perform well, you can still make money on the fees that you’re charging to manage those assets.

The rewards from those fees are so large that if you can sustain a story for why your technique is superior, you can manage assets for a long time and make a ton of money without having to perform well. And, to be fair, sometimes it takes a number of years before you know whether the quantitative technique you tried actually works or not. So even if you aren’t making money in the short term, you could have a reasonable story for why you aren’t.

At the end of the day, for the manager, it’s as important to gather a lot of assets as it is to run a successful strategy. And gathering assets can be largely a marketing game.

And you play that marketing game by talking about your algorithms and machine learning models and artificial intelligence techniques and so on.

That’s right. Let’s look at hedge funds in particular. Hedge funds are a very expensive form of investment management. So they need to justify why they’re getting paid as much as they’re getting paid.

There’s a large amount of data that suggests that the average hedge fund, after you’ve paid all the fees that they charge, is not doing much for you as an investor. The last several years in particular have not been very kind to the hedge fund industry in terms of the returns they’ve produced. So hedge funds have a strong incentive for differentiation in their marketing story. The first marketing question for a hedge fund is always, “Why are you not the average hedge fund?”

Investors want to know how a hedge fund is going to make money, given the poor performance of the hedge fund industry as a whole. These days, investors are excited by an orientation towards technology and big data and machine learning and artificial intelligence. These tools offer the promise of untapped returns, unlike older strategies that may have competed away the returns they were chasing. Regardless of whether you’re actually good at technology as a hedge fund, you want to have a story for why you might be.

Some of the most prominent hedge fund managers of the last few decades — Steve Cohen, Paul Tudor Jones — are going against type and launching technology-driven quantitative investment funds. They employ physicists and computer scientists to write algorithms to invest money, because that’s what investors want. You’re seeing a massive arms race across hedge funds to rebrand themselves in that direction.

It reminds me a bit of startup founders marketing themselves to Silicon Valley venture capitalists by peppering their pitch decks with buzzwords related to artificial intelligence or some other hot field. The startups might get funded, but the technology might not really work — or it might not even exist. What the startup is calling artificial intelligence could be a bunch of workers in the Philippines doing manual data entry.

In the financial industry, investors want firms that use big data and machine learning and artificial intelligence — but do those new tools actually generate better results?

That’s a good question. The best way to explore it might be to talk about the role of data. There’s a lot of excitement in the financial industry about the amount of new data that’s being made available. Think about what kind of data might be useful for predicting the price of an oil future. It might be a piece of political news, public announcements from regulators, satellite images of oil refineries to calculate oil reserves. There are tons of different kinds of data out there — pretty much anything you can think of.

Along with new forms of data, there are also new forms of data analysis. The early versions of complex data analysis included looking at the financial statements of publicly traded companies. But now you can parse through the data in those statements in more interesting ways. Back in the day, you might care about how much debt the company has or what its earnings are relative to its price, and you might compare those figures to the broader market. But you were ultimately limited by your capacity to source and process this data.

Now you can analyze more variables more systematically across thousands of stocks. You can also do more exotic things like use natural language processing techniques to figure out what the company is saying in its statement that isn’t reflected in its numbers. How did the commentary change from previous earnings reports? What is the tone of the words they use to describe the underlying business? How does this tone compare to words used by its competitors? Even though it’s the same data you had access to before, you have more processing power and better techniques to understand that data.

The challenge is that not all of these sources of data and ways to analyze them will be useful for predicting the prices of financial instruments. Many of the new data sets, like satellite imagery, tend to be quite expensive. And they may not add any information more useful than what is already available to market participants from the vast streams of data on prices, companies, employees, and so on. We’re still in the phase where we’re trying to figure out what to do with all the data that’s coming in. And one of the answers might be that most of it is simply not that valuable.

The big data gold rush

Let’s say it does turn out to be valuable. What does the financial industry look like then?

Everyone is competing against everyone else. If one firm succeeds in making the market more efficient through quantitative techniques, then there’s less money left over for other people to make exceptional investment returns. There will be one or two firms that are good at innovation and recognizing things that other people haven’t recognized. But everyone else will be fighting over scraps.

One of the fallacies that people have is the assumption that because the people who are working at certain firms are smart, they must be successful. But the fact that they understand artificial intelligence or machine learning or big data is somewhat useless as a competitive advantage if everyone else understands it as well.

So as those techniques get diffused across the industry as a whole, they start to be less of a differentiator. How does this impact employment? How do you see these technologies affecting either how many people the financial industry employs, or the level of skill required in different roles?

Back before the financial crisis, there was a theoretical basis for the rise of the mortgage-backed security industry. If you can diversify the risk to the investor by bundling a bunch of mortgages together, then the investor should be willing to accept a lower return, which in turn should reduce the cost to homebuyers of taking out a mortgage. That’s the theory: when financial markets work well, the benefits should percolate throughout the economy. Obviously, in 2008, that theory broke down.

In the field of quantitative investing, the same theory plays out. Let’s say people are saving money for retirement by investing in a mix of stocks and bonds. Those assets are a little cheaper for them to buy because there are all kinds of participants in the market who are fighting over making the market a little more efficient because there’s a financial incentive for them to do so.

The flip side is that the entire financial industry also has an incentive to encourage people who don’t know as much as them to give them money to do all the things that ordinary investors don’t know about. “Give me money to use a machine learning technique to manage your money, even if the machine learning technique doesn’t work, because it’s very profitable for me to take 2 percent of your fund every year.” So the incentive to make the market more efficient is balanced against the excessive proliferation of financial services that don’t add value.

What is the mechanism that’s going to eliminate that? Well, it’s the recognition that the industry as a whole may be getting paid far in excess of the value it’s providing.

How does that recognition actually begin to remake the industry, and what role will new technologies play in that process?

The short answer is that tons of jobs are on the verge of getting wiped out because technology can do those jobs. And there are benefits to scale, so you may not need many firms to replace those that don’t survive.

Take the mutual fund industry. It has more than a hundred thousand employees in the US. And every one of those jobs is at risk from the realization that the economic value of those funds is replicable with the right computer systems. For the moment, those jobs are sustained by inertia, or they are sustained by a story about why a certain manager is going to make you more money than an index fund. But that’s changing. That change will play out over the next couple of years.

Take the big money managers in Boston like Fidelity and Putnam. Those are old, large institutions. Effectively all of those jobs are at risk unless they evolve fast. And even if they do, automation will cut deep. Hedge funds, same thing. Some of them will be able to eke out value from the development of new techniques, but everyone else will be replaced by computers.

You’re already seeing big changes at investment banks. Even though investment banks continue to be very large in terms of their physical footprint, number of employees, and impact on the economy, the actual participants inside banks have changed a fair bit. It’s far more automated. Many of the actual operations inside an investment bank are done by computers. It’s not humans deciding to buy Apple stock; it’s computers deciding to buy Apple stock. So that job shift is already happening.

Financial firms are increasingly becoming tech firms. JP Morgan Chase employs 50,000 technologists, two-thirds of which are software engineers. That’s more engineers than many big tech firms: Facebook, for example, employs about 30,000 people total.

You’ve been in the financial industry for a little while so you’ve seen this transformation firsthand. How has the influx of technologists changed the industry?

The very clubby nature of traditional financial firms like investment banks has been diluted. You’ve got a lot more geeks and nerds. You don’t see certain jokes being made. Football conversations have been replaced by conversations about restaurants or other staples of yuppie culture.

The culture has mellowed quite a bit. It’s less driven by adrenaline. It’s less loud. The value is provided not by the person yelling into the phone but by the person who’s sitting at their computer, writing the right algorithm, who needs a little bit of thoughtfulness to do that work. The old model was about driving transactional flow through sheer energy. The new model is about driving transactional flow through computers.

So less “Wolf of Wall Street” and more “The Social Network.”

Totally. But that tension is still playing out. For instance, there’s still a big disconnect between the way that HR divisions recruit, especially at large firms, and the kind of candidates that are actually needed. So you’re seeing the development of completely alternative hiring tracks within large firms. The traditional hiring track just doesn’t give you enough good quantitative candidates.

Returning to the question of employment: you said you expected that one of the biggest consequences of these technologies will be a reduction in the number of people the financial industry employs. Does that also affect the overall size of the industry? On the one hand, it seems like many jobs could be eliminated or deskilled. On the other hand, it also seems possible that the very large size of the financial sector relative to the rest of the economy could be reinforced and even intensified by these technologies.

I think you’re right.

There’s a contradiction built into managing money using quantitative techniques. Let’s say you’re a hedge fund and you get paid a lot for an advanced technique. In order to demonstrate to your customer that your technique really does make money and does so in a replicable and sustainable fashion, you need to be a bit open-kimono in talking about why the technique works. You probably have to talk about the actual algorithm itself. But of course once you’ve described the algorithm, well, why does the investor need to pay a manager to do it? It’s just lines of code. Once you’ve developed it, you can run it for the marginal cost of next to zero.

Some of the managers who have been successful at raising money for their quantitative funds may have done the work of educating investors on why people shouldn’t be paying that much money to invest using these techniques. The result is that fees are dropping fast.

Currently, the largest growth in investment industry funds is happening in entities like BlackRock or Vanguard. These firms are launching a number of funds that use algorithms to invest but charge very low fees. So they are competing with hedge funds, who are having to lower their own fees in response. But BlackRock and Vanguard are also competing with themselves, because they are educating the market on why their own previous products were too highly priced.

If you measure their scale by the number of assets under management, these entities have grown at an explosive rate. BlackRock manages trillions of dollars at this point. But the actual revenue it ekes out from its assets isn’t growing nearly as fast. So you see both forces at play: the expansion of funds being managed along quantitative lines, but also the difficulty in sustaining profitability on those assets as more customers become aware of the actual cost and value of managing those assets using quantitative techniques. Even though the footprint might expand, the profitability will probably start to retreat towards levels that reflect the underlying value created.

Does that create a tendency towards greater consolidation? As the revenue that firms are able to wring out of the assets they manage goes down due to the impact of quantitative techniques, there’s presumably an incentive to manage more and more assets.

That’s certainly what we’ve seen. In the new world there are many benefits to scale. BlackRock can charge .01 percent to manage a fund because it’s got $6 trillion behind it. A smaller firm can’t compete at .01 percent. Fidelity now offers a fund that doesn’t even charge a fee. And the reason they can do that is because they have incredible scale across the rest of the platform, and if you’re a customer you might buy something somewhere else on the platform.

So consolidation is a solution to the low marginal cost of these products. Once the algorithm is known, it’s a race to see how low you can go. And how low you can go is a function of how much you manage.

Irrational cyborg exuberance

What new kind of vulnerabilities are introduced into the financial system through these techniques? What role will they play in the next financial crisis?

The way that mortgage-backed securities precipitated the financial crisis is very much applicable here. One of the fallacies behind that phenomenon was the assumption that the world would behave in the future the way it had in the past. For instance, housing prices would go ever upwards.

That fallacy is intensified in the case of quantitative investing, because all quantitative models use historical data to train themselves. As these techniques become more widespread, the assumption that the world will behave in the future the way it has in the past is being hard-wired into the entire financial system.

Another fallacy in the lead-up to the financial crisis was the assumption that financial markets were so efficient that participants didn’t need to do the underlying work to figure out what the securities were actually worth. Because you could rely on the market to efficiently incorporate all available information about the bond. All you need to think about is the price that someone else is willing to buy it from you at or sell it to you at.

Of course, if all participants believe that, then the price starts to become arbitrary. It starts to become detached from any analysis of what that bond represents. If new forms of quantitative trading rely on assumptions of market efficiency — if they assume that the price of an instrument already reflects all of the information and analysis that you could possibly do — then they are vulnerable to that assumption being false.

Is Uber worth $60 billion? Well, Uber is worth $60 billion because we believe someone is willing to pay $60 billion for it. But maybe Uber is worth zero. Maybe that’s the actual value of the revenues that Uber will make in the future. In the current environment, we rely on liquidity to sustain prices for financial assets. When liquidity dries out and you’re forced to rely on the things that those financial assets actually represent, however, you could see painful shocks if there’s a big disconnect between price and reality — the kind of shocks you saw during the financial crisis.

If people didn’t want to do the analysis before, they’re probably even less inclined to do it now. They figure the machine learning models are taking care of it.

Right. The machines are taking care of it. Or other market participants are taking care of it.

I might think that the share of a particular company is worth 20 dollars. But its price can go up to 100 dollars well before it drops down to 20, in which case I can’t sustain my measure of its actual value. So if all of the computers are pushing the price to 100 dollars, I might as well not do the work of figuring out what the company is actually worth because it’s somewhat irrelevant to the price that it trades at. Paraphrasing Keynes, “Markets can remain irrational longer than you can remain solvent.”

It sounds like algorithms have the potential to make that irrationality worse.

If the underlying computer models are less sensitive to measures of fundamental worth, they can create very large distortions in the prices of financial assets. You don’t need computers to do that, of course. You can have the Fed making a lot of cash available to everyone, cash that needs to go somewhere, and assets appreciate in response. Computers can do something similar. They can assume that prices will behave the way that their models tell them they’ll behave, and therefore drive prices to a point that is extremely disconnected from the things those prices are supposed to represent.

On February 5, 2018, the stock market fell off a cliff. The Dow industrials dropped nearly 1600 points, its worst intraday point drop in history. In the aftermath, there was a lot of discussion about the role of computerized trading in triggering the crash. Is this a preview of the world to come? Should we expect more of that in the future?

There are certainly forms of instability that have been introduced by algorithmic trading that will increase as we put more and more faith in these algorithms. The February 2018 flash crash was instructive. The culprit was a slightly esoteric exchange-traded product that has a rebalancing mechanism inside of it. And that rebalancing mechanism ended up destroying the product on one specific day when the market moved a little bit more than the product was designed to handle. The product was required to trade a lot of instruments in response to that move. But then those trades exaggerated a small move and it became a big move, which required more rebalancing — and everything spiraled out of control.

What about the impact of a more algorithmic financial system on retail investors? We’ve mostly been talking about big institutional investors, which makes sense because that’s where the money is. But how do these types of tools filter down to the ordinary investor who’s maybe got a small retirement fund?

You’ve already got robo-advisors, which use algorithms to manage assets for retail investors. We’re also probably only a few years away from you being able to log into a brokerage account and run a sophisticated institutional-grade algorithm yourself.

People tend to assume that the diffusion of these technologies is a good thing. I’m more ambivalent. I think it could be a big mistake to have the population at large play around with algorithms. Some people who are very good at it might benefit from having access to this broadened toolset. But most people will just end up paying too much or make bad decisions because they’re being given access to a technology that they aren’t equipped to do anything useful with. They can lose money with it, however.

Looking ahead, what else do you see on the horizon as finance becomes more algorithmic?

We’ve talked about the extent to which large financial firms are becoming tech firms. But I expect that it’ll start accelerating in the other direction: big tech firms will become financial firms.

If you’re a tech firm, why would you assume that a traditional financial firm is better at tech than a tech firm? If we’re talking about using big data and machine learning, well, tech firms have been doing that for a while. They’re better at data structure and organization and processing than anybody. They’re also newer, so they probably started off with better architecture internally.

In China, this is already happening. Large Chinese tech firms like Alibaba are much further along in their integration into the financial industry than their equivalents in the US. They’re doing payments and deposits and loans. The regulatory structure is more permissive. Given the expected growth of financial services there, it’s likely also a more attractive investment than for large tech companies in the US. Entrenched incumbents may be harder to dislodge here.

What financial firms have is a large customer base, which can be sticky. They also have a lot of unique knowledge — customer, economic, regulatory — from their position in the economy. But Google and Facebook have a ton of information they can employ for the same purposes — I mean, it’s hard to compete with the sheer quantity of data that tech firms have, or the scale of their integration into people’s lives. Retail investors have to put their money somewhere. They’re currently putting it into traditional financial firms. But there’s no reason that Google and Facebook shouldn’t be accepting deposits, facilitating payments, making loans, managing assets, running quantitative investment funds.

Everything I’ve described to you in the field of quantitative investing, I would imagine those companies could do very quickly. The data, the analysis, the algorithms, the infrastructure. The only question is why they haven’t yet.