In brief
Machine learning in investing is best understood as a ladder of models, from simple to complex. Peng Cheng and Emma Wu argue that linear regression remains the quant workhorse. Nonlinear models such as random forests and support vector regression are worth adopting when there is an economic reason for nonlinear behaviour, or when linear models fail out of sample. Each model has costs in interpretability and computing time.
The central tension is between power and practicality. More flexible models can capture real market patterns, such as the way equity returns relate to volatility. But they can be slow, hard to explain, or poorly suited to continuous financial data. The guests also say finance should relax its old habit of minimizing the number of variables, because modern computing power makes extra features cheap to keep.
Questions
What is the difference between AI and machine learning in investing?
Peng Cheng says machine learning is any quantitative model that produces investment signals, and it learns its own parameters from data rather than following fixed rules. AI is the broader field, which also handles text, images and voice, as ChatGPT does.
Is linear regression really machine learning?
Yes, according to Emma Wu, who calls it the fundamental machine learning algorithm. Peng Cheng adds that the difference from traditional statistics is more cultural than technical, since one field grew out of computer science and the other out of statistics.
What does overfitting mean, and how do you prevent it?
Overfitting is when a model predicts its training data well but fails on new data, which Emma compares to memorizing answers without understanding. One fix is regularization, which adds penalties to the model's scoring function so it does not fit noise.
How do random forests and boosting differ?
A random forest is a collection of decision trees, each trained on a different subset of the data, whose predictions are averaged. This is called bagging. Boosting builds models in sequence, and each new model concentrates on the mistakes of the one before it.
When should an investor choose a nonlinear model over a linear one?
Peng says to move to nonlinear models only if there is a strong economic reason to expect nonlinear relationships, or if linear models do not perform well out of sample. He also warns that tree-based models handle continuous financial relationships poorly and can be costly to run.
Facts
Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.
Peng Cheng says that, unlike a rule-based strategy, a machine learning model learns its parameters from data and finds their optimal values itself.
“But machine learning will actually learn these parameters by itself based on the data and find out what their optimal values are.”
0:02:10 · Peng Cheng · Machine learning
Peng Cheng describes machine learning as a subset of artificial intelligence.
“machine learning can be thought of as a subset of AI”
0:03:13 · Peng Cheng · AI definitions
Eloise Goulder recalls that Peng Cheng's research, discussed in a mid-September 2023 episode, found that more complex and nonlinear models tend to capture return relationships more accurately.
“the more complex the model, and indeed, if the model is non-linear, then it is much more likely to lead to a more accurate relationship”
0:04:17 · Eloise Goulder · Prior research · 2023
Emma Wu says linear regression finds the best-fit line by minimizing the sum of squared differences between actual and predicted values.
“the model aims to find the best feed line that minimizes the sum of square differences between the actual and predicted values.”
0:05:04 · Emma Wu · Linear regression
Emma Wu defines overfitting as a model that predicts its training data accurately but performs poorly on new data.
“there is a situation that the model gives accurate predictions for training data, but not for the new data.”
0:05:34 · Emma Wu · Overfitting
Emma Wu explains that regularization adds penalties to a model's loss function to help prevent overfitting.
“regularization is like you add penalties to the loss functions”
0:05:34 · Emma Wu · Regularization
Emma Wu says ElasticNet combines the lasso and ridge penalties and is useful for large datasets with many features.
“ElasticNet combines the penalties of both Lussell and Rage so that it can be useful when the data set is large and when there are large numbers of features”
0:08:47 · Emma Wu · ElasticNet
Peng Cheng says statistics has historically valued parsimony, meaning as few parameters as possible, partly because computing power was limited.
“historically, statistics have always valued parsimony, which means to have as few parameters, as few coefficients as possible in the model”
0:06:42 · Peng Cheng · Statistics culture
Peng Cheng says that 30 explanatory variables on 1,000 observations is not enough to cause overfitting.
“it's totally okay if we have a thousand observations and 30 explanatory variables.”
0:07:58 · Peng Cheng · Overfitting
Peng Cheng gives stock tracking as a use for lasso: a portfolio may be limited to about 30 stocks, so the number of variables must be restricted.
“if I want to track a stock with a portfolio of stocks, sometimes I cannot have 500 stocks tracking this one stock.”
0:10:44 · Peng Cheng · Lasso use case
Emma Wu says a random forest is essentially a collection of decision trees combined into one model.
“random forest is essentially a collection of decision tree”
0:12:28 · Emma Wu · Random forest
Emma Wu describes boosting as building models in sequence, where each new model emphasizes the errors of the previous one.
“the next model or the next member, when going into the model, consistently emphasized the error from the previous member or the model.”
0:15:07 · Emma Wu · Boosting
Emma Wu describes stacking as using the predictions of several different models as inputs to a final meta model.
“you assign a lead analyst that you trust most to take the predictions of each models as the input and make a meta model to make the final predictions.”
0:15:44 · Emma Wu · Stacking
Emma Wu's team developed a systematic tail-hedge late-cycle currency model that applies FX options selectively in risk-off markets.
“So last year, we developed a systematic tailhatch model to apply FX options selectively in risk of market and we call it a tailhatch late cycle currency model.”
0:17:13 · Emma Wu · FX tail hedge · 2022
The FX tail-hedge model began with about 30 signals spanning value, momentum, swap rates and volatility factors.
“initially we start with a list of about 30 signals across different categories such as value, momentum, swap rates, and volve factors.”
0:17:13 · Emma Wu · Feature selection · 2022
Emma Wu says that in the FX options model, linear models underperformed nonlinear ones, with higher tracking error and weaker explanatory power.
“we find that linear underperformed nonlinear models with a relatively higher tracking error and poor explanatory power.”
0:18:25 · Emma Wu · Model comparison · 2022
Peng Cheng says decision trees capture discrete nonlinear relationships well but handle continuous ones poorly.
“if you think about trees, they are pretty good at capturing nonlinear relationships, but only the discrete ones, right?”
0:19:45 · Peng Cheng · Decision trees
Emma Wu says the support vectors in a support vector model are the data points closest to the decision boundary.
“the support vector term stands for the data points that closest to the decision boundary.”
0:21:45 · Emma Wu · Support vectors
Emma Wu defines hyperparameters as settings chosen before training rather than learned from the data.
“Hyperparameters in machine learning are like settings on a recipe that you can adjust before cooking.”
0:22:18 · Emma Wu · Hyperparameters
Peng Cheng says support vector regression was developed in the 1990s, so software packages for it are easy to find and use.
“It's been developed in the 1990s, so packages are easy to find, they're easy to implement.”
0:23:27 · Peng Cheng · Support vector regression · 1990s
Peng Cheng describes a nonlinear relationship in which equity returns are good in low volatility, decline as volatility rises, and may recover at extreme volatility.
“in periods of low volatility, equity returns are probably good, right? But as volatility picks up, then equity returns declines.”
0:23:55 · Peng Cheng · Volatility and returns
Peng Cheng says a support vector machine delivered much better out-of-sample performance than several other models when predicting equity futures returns from historical volatility.
“by using SVM, It actually produces much better out-of-sample performance than a number of other machine learning models.”
0:24:26 · Peng Cheng · SVM results
Peng Cheng says support vector regression is computationally heavy and does not scale well to large problems.
“it's very computationally heavy, and it doesn't really scale well to large problems.”
0:24:26 · Peng Cheng · SVM drawbacks