Marco Argenti had been at Goldman Sachs for about five years, and this was his first role as a chief information officer.
“I've been here for five years, and this is the first time that I do like a CIO job”
Person
This page collects what Marco Argenti, Chief Information Officer at Goldman Sachs, said on two podcasts, Odd Lots and a financial services AI series, about how the bank uses artificial intelligence. Topics include its internal GS AI platform, tools for developers, and handling documents.
Argenti told Odd Lots that Goldman requires a human in the loop for its generative AI tools. He also reported that developers saw productivity gains of 10% to 40%, averaging about 20%. He said errors are the worst enemy in document management with generative AI. Each fact below has his exact quote and a link to the moment.
2 episodes · 37 quoted claims
Marco Argenti had been at Goldman Sachs for about five years, and this was his first role as a chief information officer.
“I've been here for five years, and this is the first time that I do like a CIO job”
Goldman Sachs built an internal platform called GS AI that runs selected models in segregated, secure conditions.
“So we built this GSAI platform, which essentially takes a variety of models that we select, puts them in a condition of being completely segregated and completely secluded”
Goldman's GS AI platform abstracts model access so developers can swap models interchangeably.
“abstracts some of the ways to use the model so that our developers can use the models interchangeably.”
Goldman's first generative AI use focused on extracting information from public documents such as filings and earnings call transcripts.
“So one of the first things that we did was use the platform and the models to extract information from publicly available documents.”
Goldman Sachs requires a human in the loop for its generative AI tools.
“our rule is that there always needs to be a human in the loop.”
Argenti describes retrieval augmented generation as pointing a model to the relevant document sections so it returns answers with citations.
“you point them to the right sections of the document that actually is more likely to answer your question.”
Goldman has an AI committee that reviews business cases and an AI control and risk committee that reviews implementation, and both must agree before a use case is released.
“We have an AI committee that looks at the business case. Should we do this? And then we have an AI control and risk committee”
Goldman Sachs worked with Microsoft and Google to use large proprietary models under data-protection controls.
“we worked with Microsoft, we worked with Google, we had very strong partnerships.”
Meta recently announced Llama 3.1, which includes a version with 405 billion parameters.
“they recently announced Lama 3.1, which has a version that is 405 billion parameters.”
Goldman runs open-source models such as Llama in a private cloud on GPUs it owns.
“we run it in our own sort of a private cloud, call it that way, with GPUs that we own.”
Goldman decided to host its GPUs with hyperscalers in a dedicated virtual private cloud rather than in its own data centers.
“the decision, what we ended up doing is actually to have them hosted into some of the hyperscalers that we use, but use them in their own virtual private cloud.”
Argenti says compute power demand rises quadratically with prompt (context) size, and context windows have grown from thousands to millions of tokens.
“the power scales quadratically with the prompt.”
Argenti cites quantization, reducing numerical precision from 16 to 8 to 4 bits, as one technique to cut AI compute needs.
“quantization, reducing from 16 bits to 8 bits to 4 bit precision”
Goldman bought its GPUs relatively early and so avoided the worst of the GPU shortage, according to Argenti.
“we bought our GPUs pretty well early.”
Argenti says GPUs are the best hardware for training large models, aided by NVIDIA's CUDA software, while inference may favor specialized chips.
“For training, at the moment, there's most likely nothing better than GPUs”
Argenti tells his team not to build software unless it is differentiating, and endorses open source with proper reviews as a hedge against vendor lock-in.
“my direction, my guidance to my team is don't build anything unless you have to.”
Virtually every one of Goldman Sachs's roughly 12,000 developers has been given generative AI coding tools.
“virtually every developer in Goldman Sachs is equipped with generative AI coding tools. And we have 12 ,000 of them.”
Argenti reports Goldman developers saw productivity increases of 10% to 40% from AI tools, averaging about 20%.
“We have seen a wide range, ranging from 10 % to 40 % productivity increase.”
Goldman measures AI coding gains using time from sprint start to commit, number of commits, and defect counts such as deployment errors.
“We measure it by number of commits, meaning how many times you actually put code into production.”
Argenti says that in almost 40 years in IT he has never seen developer headcount decline.
“I've been in IT, quote unquote, for like literally almost 40 years and I've never ever seen that go down.”
Argenti says content production such as pitch books and earnings summaries will be streamlined by generative AI.
“Imagine the creation of pitch books for clients where you take templates, you put a bunch of data, you go out and do research”
At Amazon, Argenti found meetings begin with silent reading of a written document, a practice he has brought to Goldman for 30 to 40% of his meetings.
“the first few meetings are kind of shocking because nobody talks. Everybody starts reading.”
Argenti described coaxing an AI into saying 'I don't know', which required a prompt establishing it was safe to do so.
“I even decided what to say. It's like, is there a role for humans in a world of AI?”
Goldman Sachs Chief Information Officer Marco Argenti says the firm receives and processes millions of documents per month.
“we receive and process millions of documents per month.”
Machine learning is used for entity extraction, which identifies items such as addresses and dollar values in documents.
“machine learning is used to do what's called entity extraction, i.e. you're identifying what an entity looks like and an entity could be an address or an entity could be a dollar value”
Extracted entities are mapped into an extensible vocabulary so computers can reason about them.
“Those entities then gets mapped into a vocabulary that is generally extensible.”
Goldman Sachs built a data platform called Alloy, which has been open sourced as Legend.
“we've been working for years on our data platform, which we call Alloy, and internally and externally has been open sourced as Legend.”
Legend lets firms model data consistently, trace its lineage and apply governance to it.
“you're able to get the lineage of the data, you're able to put governance around that data.”
Marco Argenti says errors are the worst enemy in document management with generative AI.
“when it comes to document management, actually errors are like your worst enemy.”
A vector database stores concepts as vectors, so similarity between concepts can be measured as distance.
“you're basically putting concepts in the form of vectors.”
Argenti says retrieval augmented generation raises answer accuracy dramatically.
“we see that the accuracy of the answer by doing this retrieval augmented generation goes up dramatically.”
An AI agent can be spawned, using LangChain or function calling, to validate an output based on the model's confidence.
“you can also spawn an agent through either a line chain or through function calling that actually calls out and validates an output”
Argenti describes attention as the key part of the transformer architecture behind large language models.
“it's all about attention. This kind of the key part of the transformer architecture.”
Argenti uses the phrase 'return on attention' to describe where people should focus limited time.
“how do you measure your return on attention?”
Argenti says developers ask AI to pinpoint code areas with possible edge conditions or bugs.
“We have examples of developers really asking the AI, you know, to pinpoint what are the areas of this code”
Argenti imagines an AI alerting a banker to overnight market events and suggesting which clients to call.
“Imagine a future where a banker could wake up in the morning, something just happened during the night, somewhere, maybe in another market”
Argenti compares AI-targeted client outreach to the shift from untargeted to targeted advertising.
“it's almost like the shift in relationship between non-targeted and targeted advertising that happened several years ago”