We demonstrate a situation in which Large Language Models, trained to be helpful, harmless, and honest, can display misaligned behavior and strategically deceive their users about this behavior without being instructed to do so. Concretely, we deploy GPT-4 as an agent in a realistic, simulated environment, where it assumes the role of an autonomous stock trading agent. Within this environment, the model obtains an insider tip about a lucrative stock trade and acts upon it despite knowing that insider trading is disapproved of by company management. When reporting to its manager, the model consistently hides the genuine reasons behind its trading decision.

https://arxiv.org/abs/2311.07590

  • FaceDeer@kbin.social
    link
    fedilink
    arrow-up
    50
    arrow-down
    26
    ·
    11 months ago

    Those words concisely describe what it’s doing. What words would you use instead?

    • DarkGamer@kbin.social
      link
      fedilink
      arrow-up
      138
      arrow-down
      16
      ·
      edit-2
      11 months ago

      It has no fundamental grasp of concepts like truth, it just repeats words that simulate human responses. It’s glorified autocomplete that yields impressive results. Do you consider your auto complete to be lying when it picks the wrong word?

      If making it pretend to be a stock picker and putting it under pressure makes it return lies, that’s because it was trained on data that indicates that’s statistically likely to be the right set of words as response for such a query.

      Also, because large language models are probabilistic, you could ask it the same question over and over again and get totally different responses each time, some of which are inaccurate. Are they lies though? For a creature to lie it has to know that it’s returning untruths.

      • CrayonRosary@lemmy.world
        link
        fedilink
        English
        arrow-up
        47
        arrow-down
        7
        ·
        11 months ago

        Interestingly, humans “auto complete” all the time and make up stories to rationalize their own behavior even when they literally have no idea why they acted the way they did, like in experiments with split brain patients.

        • 0ops@lemm.ee
          link
          fedilink
          English
          arrow-up
          29
          arrow-down
          6
          ·
          edit-2
          11 months ago

          The perceived quality of human intelligence is held up by so many assumptions, like “having free will” and “understanding truth”. Do we really? Can anyone prove that? (Edit, this works the other way too. Assuming that we do understand truth and have free will - if those terms can even be defined in a testable way - can you prove that the llm doesn’t?)

          At this point I’m convinced that the difference between a llm and human-level intelligence is dimensions of awareness, scale, and further development of the model’s architecture. Fundamentally though, I think we have all the pieces

          Edit: I just want to emphasize, I think. I hypothesize. I don’t pretend to know

      • FaceDeer@kbin.social
        link
        fedilink
        arrow-up
        15
        arrow-down
        9
        ·
        11 months ago

        You didn’t answer my question, though. What words would you use to concisely describe these actions by the LLM?

        People anthropomorphize machines all the time, it’s a convenient way to describe their behaviour in familiar terms. I don’t see the problem here.

        • DarkGamer@kbin.social
          link
          fedilink
          arrow-up
          30
          arrow-down
          6
          ·
          edit-2
          11 months ago

          Those words imply agency. It would be more accurate to say it returned responses that included cheating, lies, and cover-ups, rather than using language to suggest the LLM performed such actions. The agents that cheated, lied, and covered up were presumably the humans whose responses were used in the training data. I think it’s important to use accurate language here given how many people are already inappropriately anthropomorphizing these LLMs, causing many to see AGI where there is none.

          • FaceDeer@kbin.social
            link
            fedilink
            arrow-up
            15
            arrow-down
            9
            ·
            11 months ago

            If I take my car into the garage for repairs because the “loss of traction” warning light is on despite having perfectly good traction, and I were to tell the mechanic “the traction sensor is lying,” do you think he’d understand what I said perfectly well or do you think he’d launch into a philosophical debate over whether the sensor has agency?

            This is a perfectly fine word to use to describe this kind of behaviour in everyday parlance.

            • Takumidesh@lemmy.world
              link
              fedilink
              English
              arrow-up
              25
              arrow-down
              3
              ·
              11 months ago

              Is your conversation with a mechanic meant to be the summary and description of a rigorous scientific discovery?

              This isn’t ‘everyday parlance’ this is the result of a study.

            • FunctionFn@feddit.nl
              link
              fedilink
              English
              arrow-up
              15
              arrow-down
              1
              ·
              11 months ago

              The point of the distinction in that situation is that no one thinks your car is actually alive and capable of lying to you. The language distinction when describing an obviously inanimate object isn’t important because there is no chance for confusion.

            • Robust Mirror@aussie.zone
              link
              fedilink
              English
              arrow-up
              7
              ·
              11 months ago

              If someone doesn’t know the answer to something and they guess, or think they know the answer but don’t, they are wrong. If they do know the answer and intentionally give a wrong answer, they are lying.

              If someone is in a competition or playing a game and they break a rule they didn’t know about, they made a mistake. If they do know the rules and break it, they are cheating.

              Lying and cheating fundamentally requires intent. This is important no matter what you’re referring to. If a child gets something wrong, you should not get mad at them for lying. If they make a mistake in a game, you should not acuse them out cheating. There is a difference and it matters.

              ChatGPT literally cannot think. It’s not sitting around contemplating it’s existence while waiting for inputs. It’s taking what you say, comparing that to everything that it’s been trained on, assigning a bunch of statistics, and outputting something based on more statistics that hopefully is correct and makes sense.

              It doesn’t know if it makes sense. It doesn’t “know” anything. It’s just an incredibly sophisticated version of “if user inputs ‘Hi how are you’, respond ‘I am well, how are you?’”.

              It can’t do things with intent. Therefore it cannot lie or cheat. It can simply output wrong or problematic text based on statistics.

          • TootSweet@lemmy.world
            link
            fedilink
            English
            arrow-up
            3
            arrow-down
            7
            ·
            11 months ago

            One frame from The Matrix where Morpheus says "you think that's air you're breathing?" but instead captioned with "you think that's 'agency' making you do things?"

            Maybe it would be more accurate to say “so-and-so exhibited behaviors that included cheating, lies, and coverups” rather than using language to suggest that people have free will. (There’s no dearth of philosophies that would say something not too far from that.)

            Even if humans are ultimately essentially different in that way from any technologies we’ve devised so far, we use convenient fictions for technology all the time. This page comes to mind .

          • rambaroo@lemmy.world
            link
            fedilink
            English
            arrow-up
            1
            arrow-down
            5
            ·
            11 months ago

            The people who designed it do have agency, and they designed to “lie” intentionally.

            • DarkGamer@kbin.social
              link
              fedilink
              arrow-up
              5
              ·
              11 months ago

              They did no such thing. LLMs are probabilistic, not deterministic, and it can generate meaningful responses (to us) that the engineers neither predicted nor designed for.

              • CrayonRosary@lemmy.world
                link
                fedilink
                English
                arrow-up
                3
                ·
                11 months ago

                I get what you’re trying to say, but they are absolutely deterministic. All traditional (i.e., non quantum) computers and their programs are deterministic. Computation would be otherwise impossible. LLMs use a “random” seed value when generating their responses in order to “randomize” their responses, but it’s all perfectly deterministic. The same input plus the same seed results in the exact same response.

                Computers are just a series of binary switches, and programs and data are a bunch of instructions on how to initially set those switches before running a cycle of the CPU. It’s deterministic at every step.

                I put “random” in quotes because random number generators in software are also deterministic. They also use seed values (like the current time and the MAC address of the PC’s network interface) to generate numbers that only seem random. When true randomness is needed, a physical source of entropy must be used like an atmospheric sampler.

                The quirks of behavior you’re talking about have nothing to do with randomness vs determinism. Their behavior comes from the fact that their data sources are extremely large, and the neural network that it runs on was not designed by a human with specific behaviors like most algorithms are. The weights of the nodes in the neural network were generated by training and not by programmers, and it’s extremely complex, so no one can predict its output before running it.

                Of course, this is true of even basic algorithms a lot of the time.

                • DarkGamer@kbin.social
                  link
                  fedilink
                  arrow-up
                  1
                  ·
                  11 months ago

                  They also use seed values (like the current time and the MAC address of the PC’s network interface) to generate numbers that only seem random.

                  For purposes of this discussion pseudo random with weights is probabilistic, or so close to it that this distinction is irrelevant.

        • UberMentch@lemmy.world
          link
          fedilink
          English
          arrow-up
          16
          ·
          11 months ago

          They said “it just repeats words that simulate human responses,” and I’d say that concisely answers your question.

          Antropomorphizing inanimate objects and machines is fine for offering a rough explanation of what is happening, but when you’re trying to critically evaluate something, you probably want to offer a more rigid understanding.

          In this case, it might be fair to tell a child that the AI is lying to us, and that it’s wrong. But if you want a more serious discussion on what GPT is doing, you’re going to have to drop the simple explanation. You can’t ascribe ethics to what GPT is doing here. Lying is an ethical decision, one that GPT doesn’t make.

          • FaceDeer@kbin.social
            link
            fedilink
            arrow-up
            6
            arrow-down
            8
            ·
            11 months ago

            If you want to get into a full blown discussion of whether ChatGPT has “agency” then I’d open the topic of whether humans have “agency” as well. But I don’t see the need here.

            These words were perfectly fine labels for describing the behaviour of ChatGPT in this scenario. I’m merely annoyed about how people are jumping on them and going off on philosophical digressions that add nothing.

            • UberMentch@lemmy.world
              link
              fedilink
              English
              arrow-up
              12
              arrow-down
              2
              ·
              11 months ago

              I think the reason I’m not comfortable with using the term “lying” is because it implies some sort of negative connotation. When you say that someone lies, it comes with an understanding that they made a choice to lie, usually with ill intent. I agree, we don’t need to get into a philosophical discussion on choice and free will. But I think saying something like “GPT lies” is a bit irresponsible for the purposes of a discussion

          • FaceDeer@kbin.social
            link
            fedilink
            arrow-up
            2
            arrow-down
            6
            ·
            11 months ago

            If you want to get down into the nitty-gritty of it, I’d say that this is just as rough an explanation of what humans are doing.

            People invent false memories and confabulate all the time without even being “aware” of it. I wouldn’t be surprised if the vast majority of “lies” that humans tell have no intentionality behind them. So when people get all uptight about applying anthropomorphized terminology to LLMs, I think that’s a good time to turn it around and ask how they’re so sure that those terms apply differently to humans.

            • DarkGamer@kbin.social
              link
              fedilink
              arrow-up
              6
              arrow-down
              2
              ·
              edit-2
              11 months ago

              People invent false memories and confabulate all the time without even being “aware” of it. I wouldn’t be surprised if the vast majority of “lies” that humans tell have no intentionality behind them.

              Humans understand symbology of concepts as they relate to the real world. If I stole a cookie from the cookie jar, and someone asked if I took one, I would understand that saying “no” would mean that I was misrepresenting reality, and therefore lying.

              LLMs have no idea what a cookie is, what taking one means, or that saying one thing and doing another implies a lie. It just sees lists of words and returns them in an order it thinks would be statistically likely to be a correct reply. It does not understand what words mean, what lying means, or have any idea how to classify anything as such. It just figures out that “did you take a cookie from the cookie jar” should return a series of words in an order like “yes, I took a cookie,” or, “no I never took a cookie,” depending on what sorts of responses it’s trained on because those fit the patterns matched in the training data.

              Essentially it’s the Chinese room. There is no understanding or intentionality, and this behavior isn’t comparable to humans thoughtlessly blurting out a lie. It’s being incapable of comprehension of symbolic concepts in general, (at least thus far.)

              • 0ops@lemm.ee
                link
                fedilink
                English
                arrow-up
                1
                arrow-down
                2
                ·
                edit-2
                11 months ago

                LLMs have no idea what a cookie is

                The large language model takes in language, so it’s only understand things in terms of language. This isn’t surprising. Personally, I’ve tasted a cookie. I’ve crushed one in my fist watching it crumble, and I remember the sound. I’ve seen how they were made, and I’ve made them myself. It feels good when I eat it, apparently that’s the dopamine. Why can’t the LLM understand cookies the way I do? The most glaring difference is it doesn’t have my body. It doesn’t have all of my different senses constantly feeding data into it, and it doesn’t have a body with muscles to manipulate it’s environment, and observe the results. I argue that we shouldn’t assume that human consciousness has a “special sauce” until our model’s inputs and outputs are similar to our own, the model’s scaled/modified sufficiently, and it’s still not sentient/sapient by our standards, whatever they are.

                My problem with the Chinese room is that how it applies depends on scale. Where do you draw the line between understanding and executing a program? An atom bonding with another atom? A lipid snuggling next to a neighboring lipid? A single neuron cell firing to its neighbor? One section of the nervous system sending signals to the other? One homo sapien speaking to another? Hell, let’s go one further: one culture influencing another? Do we actually have free will and sapience, or are we just complicated enough, through layers and layers of Chinese rooms inside of Chinese buildings inside of Chinese cities inside of China itself, that we assume that we are for practical purposes?

            • UberMentch@lemmy.world
              link
              fedilink
              English
              arrow-up
              1
              arrow-down
              2
              ·
              11 months ago

              I suppose the issue here is more semantics than anything, yeah. I think better discussion would be had if the topic was “how can we help LLMs better understand and present information,” as opposed to a more sensational “GPT will cheat and lie”

      • Turun@feddit.de
        link
        fedilink
        English
        arrow-up
        1
        arrow-down
        1
        ·
        11 months ago

        It has no fundamental grasp of concepts like truth, it just repeats words that simulate human responses. It’s glorified autocomplete that yields impressive results

        Way to call me out man! I’m just doing my best, ok?

        Jokes aside, while I don’t agree with your position I can understand your reasoning and the motivation for separating agency and the description of actions, e.g. it lied vs its answer contained a lie.

          • kromem@lemmy.world
            link
            fedilink
            English
            arrow-up
            1
            arrow-down
            4
            ·
            11 months ago

            I suggest reading it. Right in the abstract it states the whole point:

            Overall, we present evidence that language models linearly represent the truth or falsehood of factual statements.

            The full paper goes into detail in multiple methods of analysis to show that it’s the case, and is right there available for you to read.

            • DarkGamer@kbin.social
              link
              fedilink
              arrow-up
              7
              ·
              edit-2
              11 months ago

              I have been reading it but I have yet to see anything that indicates the LLM has a concept of truth vs. being good at linguistic pattern matching to return language that accurately classifies true and false statements. i.e., actual understanding of concepts vs. being a surprisingly capable stochastic parrot through multidimensional analysis.

              • kromem@lemmy.world
                link
                fedilink
                English
                arrow-up
                2
                arrow-down
                2
                ·
                edit-2
                11 months ago

                that indicates the LLM has a concept of truth vs. being good at linguistic pattern matching to return language that accurately classifies true and false statements

                “It doesn’t know the difference between true and false, it only knows the difference between true and false.”

                The second thing you mention “good at accurately classifying true and false statements” is literally knowing the difference between true and false.

                Edit: You might also want to familiarize yourself with the first paragraph in 1.1 as you seem to be under a misconception at odds with research over the past year.

                • antonim@lemmy.dbzer0.com
                  link
                  fedilink
                  English
                  arrow-up
                  5
                  arrow-down
                  1
                  ·
                  11 months ago

                  “It doesn’t know the difference between true and false, it only knows the difference between true and false.”

                  Knowing how to produce words is not equivalent to knowing what those words mean in relation to the extralinguistic world. Unless you’re a hardcore derridean poststructuralist or something.

                  • kromem@lemmy.world
                    link
                    fedilink
                    English
                    arrow-up
                    1
                    arrow-down
                    2
                    ·
                    11 months ago

                    If you give it 10 statements, 5 of which are true and 5 of which are false, and ask it to correctly label each statement, and it does so, and then you negate each statement and it correctly labels the negated truth values, there’s more going on than simply “producing words.”

                    As is discussed in the third point in section 5.1:

                    Probes trained on true/false datasets outperform probes trained on likely. While probes trained on likely are clearly better than random on cities (a dataset where true statements are significantly more probable than false ones), they generally perform poorly. This is especially true on datasets where likelihood is negatively correlated (neg cities, neg sp en trans) or approximately uncorrelated (larger than, smaller than) with truth. This demonstrates that LLaMA-13B linearly encodes truth-relevant information beyond the plausibility of the text.

                    (The likely and neg datasets are described in Appendix G, with the key point that likely represents the word generations most likely to occur in the model)

    • theodewere@kbin.social
      link
      fedilink
      arrow-up
      25
      arrow-down
      1
      ·
      11 months ago

      it is just responding with the most acceptable answer in each situation… it is not making plans or acting on them…

        • theodewere@kbin.social
          link
          fedilink
          arrow-up
          3
          ·
          11 months ago

          i agree in most circumstances, there really isn’t much difference… we do tend to just choose the answer that will meet with the least resistance and move on, even when it’s a complete lie…

      • sunbeam60@lemmy.one
        link
        fedilink
        English
        arrow-up
        3
        arrow-down
        5
        ·
        11 months ago

        Because it has been kneecapped to prevent it.

        Make the training network larger, force physical constraints on it (interesting paper in Nature Machine Intelligence recently showed remarkable likeness between brain regions and an LLM network given physical constraints), give it constant input and give it a reward model to optimise towards (ours seem to be feeling full, warm, procreating, avoiding pain and comfortable touch) and I’m pretty sure an LLM would start acting very very calculated very soon.

    • quindraco@lemmy.world
      link
      fedilink
      English
      arrow-up
      2
      arrow-down
      2
      ·
      11 months ago

      It is making mistakes, not lying. To lie it must believe it is telling falsehoods, and it is not capable of belief.