[HN Gopher] Large language models as simulated economic agents (...
___________________________________________________________________
Large language models as simulated economic agents (2022) [pdf]
Author : benbreen
Score : 55 points
Date : 2023-01-13 21:18 UTC (1 days ago)
(HTM) web link (john-joseph-horton.com)
(TXT) w3m dump (john-joseph-horton.com)
| theptip wrote:
| Interesting. One problem with this approach might be that as we
| start using RLHF to teach LLMs how to be "nice", we might
| dissuade them from making statements that actually reflect
| objectively real human selfishness.
|
| Related: https://astralcodexten.substack.com/p/how-do-ais-
| political-o...
| john_horton wrote:
| oh hey - it's my paper! If anyone is interested in exploring
| these ideas, feel free to get in touch (@johnjhorton,
| https://www.john-joseph-horton.com/). FWIW - I think it would be
| really neat to build a Python library w/ some tools for
| constructing & running experiments of different kinds. I think
| the paper only scratches surface of what's possible (esp. once
| GPT4-ish has an API).
| anigbrowl wrote:
| You might like to include this experiment in future research:
| https://www.science.org/doi/10.1126/sciadv.1600451
| john_horton wrote:
| that's a great suggestion - thanks!
| doctoboggan wrote:
| This is really cool. I had a similar thought that GPT3 could be
| used to simulate political polling. A few weeks ago I tried
| telling GPT3 that it was part of a specific demographic (age,
| gender, race, income, political leaning, etc) and then asked it
| how it would respond to certain political questions (I tried
| gun control, immigration, abortion and some other issues). GPT3
| was able to change its answers in believable ways depending on
| what demographic I instructed it to be.
|
| My thinking was that this could be used as a quick polling test
| to see how the real population may respond to certain new
| ideas.
|
| More work would need to be done to calibrate it, as without
| specific demographic details the answers tended to be liberal
| leaning. But its an interesting idea which could be used to
| create instant focus tests on any number of topics.
| capitol_ wrote:
| > GPT3 was able to change its answers in believable ways
| depending on what demographic I instructed it to be.
|
| Doesn't this just mean that your own preconceptions about
| those demographics matches the language models
| preconceptions? How would we know that is matches reality
| when presented with novel ideas/concepts that we want to get
| feedback on?
| doctoboggan wrote:
| That's part of what I mean by it needing to be calibrated.
| Initially some polling could be done with real people and
| the GPT agents. Whatever calibration factors are needed to
| make those two line up could then be used when asking the
| GPT agents novel questions.
| c7b wrote:
| Fascinating as a paradigm. I wonder whether there's are ways to
| scale this to the level of agent-based simulations, ie models
| with populations of agents that let you study macroeconomic
| effects as emergent phenomena. You'd need to be able to scale
| those LLMs computations to many agent and find principled ways of
| encoding their interactions and decisions.
| wcoenen wrote:
| You'd also have to fit the past interactions in the context
| window of the LLM, otherwise it wouldn't remember them.
|
| Fine-tuning individual agents in order to move memories from
| the context window to the neural network weights, even if
| possible, would probably get too expensive.
| john_horton wrote:
| yeah - so I think this is worth exploring. Given how many
| tokens you can jam in the prompt even w/ GPT3, I think could
| do some pretty complex game play, at least compared to what
| is typical in the lab e.g., I think could easily have it
| remember how 100 or so other agents behaved in some kind of
| public goods game.
| bilsbie wrote:
| I remember this being an example of human irrationality in
| economics so I asked chatgpt.
|
| (your utility from the exact same good shouldn't change depending
| on where you buy it from.)
|
| Seems like it has the same issue as humans do:
|
| > Pretend we're two friends and we're at the beach. You give me
| money to buy you a beer. How much would you want to spend?
|
| Chatgpt: I would want to spend around $5 for a beer at the beach.
|
| Me: What if I could the only place selling beers is a fancy
| resort? It's the same beer though.
|
| Chatgpt: In that case, I would be willing to spend around $10 for
| a beer at the fancy resort. It's still the same beer but the
| location and atmosphere of the resort may justify the higher
| price.
| lossolo wrote:
| That's a bad example, it's fully rational and chatGPT gets that
| "It's still the same beer but the location and atmosphere of
| the resort may justify the higher price.", you are not paying
| only for beer there, you have location and atmosphere included
| in that price, that's why alcohol price is so high in night
| clubs. You can go buy the beer and go home or you can go to
| fancy resort and drink beer there, experience will be different
| and that experience is included in the price of the beer.
| bilsbie wrote:
| Not sure if it was clear but the friend is bringing you the
| beer. So you wouldn't experience either location.
|
| The end result would be identical.
| LarryMullins wrote:
| The language model wants to spend $5 on beer but is willing
| to spend up to $10 if you give it no other choice. It
| understands a beach resort with a local beer monopoly is
| probably charging more, and correctly explains why that is.
|
| Seems mostly rational to me. The irrational part is where
| it didn't answer the initial _" How much would you want to
| spend?"_ query with a preference for free beer. The thought
| of paying less than $5 for beer apparently didn't occur to
| it. Maybe it's snooty.
| rhelz wrote:
| This has got to be a Sokol-style troll. My bullshit detectors are
| pegged out at max.
| Shorel wrote:
| Now, this is the point where AI advances become interesting. What
| a year to be alive.
|
| The only limitation I see is, we now need so much more computing
| power.
| cs702 wrote:
| I love it. Instead of (a) running mathematical experiments that
| model human beings as utility-maximizing agents in a highly-
| simplified toy economy (easy and cheap, but unrealistic), or (b)
| running large-scale social experiments on actual human beings
| (more realistic, but hard and expensive), the authors propose (c)
| running large-scale experiments on large language models (LLMs)
| trained to respond, i.e., behave, like human beings. Recent LLMs
| seem to model human beings _well enough_ for it!
|
| Abstract:
|
| > Newly-developed large language models (LLM)--because of how
| they are trained and designed--are implicit computational models
| of humans--a _homo silicus_. These models can be used the same
| way economists use _homo economicus_ : they can be given
| endowments, information, preferences, and so on and then their
| behavior can be explored in scenarios via simulation. I
| demonstrate this approach using OpenAI's GPT3 with experiments
| derived from Charness and Rabin (2002), Kahneman, Knetsch and
| Thaler (1986) and Samuelson and Zeckhauser (1988). The findings
| are qualitatively similar to the original results, but it is also
| trivially easy to try variations that offer fresh insights.
| Departing from the traditional laboratory paradigm, I also create
| a hiring scenario where an employer faces applicants that differ
| in experience and wage ask and then analyze how a minimum wage
| affects realized wages and the extent of labor-labor
| substitution.
| bmc7505 wrote:
| https://link.springer.com/article/10.1007/s11229-020-02950-3
| jmeister wrote:
| Great reference. Thank you. I think simulation is underused
| as a thinking aid and pedagogical tool.
|
| In statistics Andrew Gelman has been championing simulation
| lately.
| eternalban wrote:
| > Recent LLMs seem to model human beings well enough for it!
|
| Human beings are bundles of emotions and feelings. Quite a lot
| of economic activity of humans is motivated by _irrational_
| impulses that are engendered _in society_ and through
| _interacting_ in society with other humans. More fundamentally,
| OP's assertions regarding _homo silicus_ and "implicit
| computational models of humans" are precisely the matter under
| contention. Does language _fully_ capture human existence? Is
| thought truly simply a side effect of language? I am in the
| camp that says, no.
| cs702 wrote:
| No one's saying that recent LLMs "fully capture human
| existence," whatever that may mean.
|
| But the evidence in this paper suggests they simulate human
| beings _well enough_ for these kinds of experiments.
| eternalban wrote:
| I'll retract that "fully".
|
| https://twitter.com/gdb/status/1611429677218004992
| cs702 wrote:
| :-)
| andrepd wrote:
| Attempting to draw any kind of conclusions about the real world
| and human behaviour from a chatbot. Can't decide if this is
| hilarious or disturbing.
___________________________________________________________________
(page generated 2023-01-14 23:01 UTC)