嗯。
Yeah.
说真的,如果坐上时光机回到过去,那时我就已经在自己的卧室里,用一台 8 位电脑编写 AI 软件了。
I mean, really, when you go back in the time machine, I was already building AI software in my bedroom on an 8-bit computer.
我的第一台是雅达利 400,只有 16K 的 RAM,我同时用机器语言和 Atari BASIC 来编程。
My first one, Atari 400, with 16K of RAM and using both machine language as well as Atari BASIC.
有意思的是,我基本上做出了一个能自我修改的学习模拟程序。
And, you know, basically, I built, interestingly, I built the self-modifying learning simulation.
所以我那时做的,算是认知主义流派的机器学习,大概是在上世纪 80 年代读高中的时候。
So, I was doing kind of cognitivist machine learning back then, and this was kind of in high school, back in the 1980s.
真的是非常非常早,回到了个人计算机刚刚起步的年代。
So, really, way, way back in the origins of the beginnings of personal computing.
说来好笑,我做了一个游戏,叫“动物猜猜看”(Animal Guesser)。
And so, you know, I built, it's funny, I built this game that I call Animal Guesser.
它基本上是玩“二十个问题”的游戏,机器通过一连串“是或否”的问题来猜你想的是哪种动物。
And it would basically try to take 20 questions, and the machine would try to guess the animal based on yes or no questions.
它只会问你“是”或“否”的问题,对吧?
And it would just ask you yes or no questions, right?
它其实非常原始,因为它本质上就是在做二分查找。
And it was really primitive, because what it would do is it would basically do a binary search.
也就是说,它会尽力每次把剩下的搜索空间砍掉一半,对吧?
So, it would basically make a best effort to cut the remaining search space in half each time, right?
它会去查看自己已知的那份可能性清单,对吧?
So, you know, so it would look at the list of possibilities that it knew about, right?
但我当时觉得它真正的创新之处在于,这个程序用上了 Atari BASIC 里一个奇特的“后门”,也就是自我修改代码。
But the thing that I felt like was innovative about this at the time was that the program actually used this weird escape hatch in the Atari BASIC, which was self-modifying code.
结果就是,每一轮结束时,这段代码,这个软件,真的会重写它自己。
So, what happened is that the code would actually, the software would actually rewrite itself at the end of each round.
因为一旦它猜输了,它就会说:我不知道长颈鹿是什么。
Because what would happen is, is that if it would ever lose, it would be like, I don't, you know, like, I don't know what a giraffe is.
它会问:你能不能帮我想一个“是或否”的问题,把它和斑马区分开来?
Like, could you create me a yes or no question that would distinguish this from a zebra?
诸如此类吧,然后它基本上就会把这个记下来。
Or whatever, like, whatever, you know, and so what it would do is it would basically memorize that.
长话短说,我在 1990 年写的那个连接主义的误差反向传播网络,并不是我第一次接触机器学习。
So, you know, long story short, like, you know, the connectionist error backpropagation network I wrote in 1990 was not my first machine learning.
显然,那些早期的机器学习大多是我自学的,真的差不多就是个孩子在玩耍而已,对吧?
Like, you know, but obviously, a lot of that early machine learning was sort of self-taught and, you know, really just almost like a kid just playing, right?
就是很单纯地异想天开:我能不能让计算机做点有意思的事?
So, just really trying to dream up, can I make the computer do something interesting, right?
所以那些都是我在很小的年纪就自己捣鼓出来的,对吧?
So, that was, you know, doing it on my own, you know, at this pretty early age, right?
但后来真正发生的,是连接主义革命,以及复杂性理论等思潮的兴起。
But, you know, eventually, what happened was really the connectionist revolution, and it was the rise of things like complexity theory.
1986 年那篇重磅论文出自鲁梅尔哈特和辛顿,也就是那篇误差反向传播的论文,对吧?
And, you know, in 1986, the big paper was Rumelhart and Hinton, which is the error backpropagation paper, right?
那是一篇非常重要的论文。
So, that was a really big paper.
所以到了 1990 年,我们都纷纷加入了这股潮流。从某种意义上说,鲁梅尔哈特和辛顿当时真正推动的,是一场变革:让神经科学从认知主义模型转向连接主义模型,对吧?
So, you know, by 1990, we were all jumping on the bandwagon, you know, and, you know, in a sense, what Rumelhart and Hinton at the time were really pushing, was a revolution away from the cognitivist model of neuroscience and towards a connectionist model, right?
而真正有意思的是,当我们进入现代 AI 时代,令人着迷的一点在于:GPT 这类系统的核心,依然是同样的机器学习误差反向传播。
So, the thing that's so interesting becomes that, as we come into the modern era of AI, it's fascinating because the heart of GPT class systems is the same machine learning error backpropagation.
但与此同时,新加入的东西,当然,就是 Transformer,这一点意义重大,对吧?
But, at the same time, what's added, of course, what's added is transformers, which is huge, right?
所以机器的“注意力”机制,是那个年代以来最大的变化。
So, machine attention was the biggest change since those days.
但与此同时,我认为真正凸显我们今天所处阶段的一点,而这一点其实广为人知却又鲜被提及,就是:那些鼓吹持续学习之类概念的人,往往执迷于“灾难性遗忘”这个想法,也非常执迷于所谓“神经可塑性内核”的概念。
But, at the same time, the thing that really, I think, accentuates where we are today, I think that's very understood and underrepresented is that people who are touting things like continuous learning are often obsessed with this idea of, like, catastrophic forgetting, and they're really obsessed with this idea of, like, a neuroplastic core.
他们如此痴迷于神经可塑性,是因为大脑本身在本质上就是可塑的。
So, they're really obsessed with neuroplasticity because the brain itself is essentially neuroplastic.
不过我要说,大脑的神经可塑性当中,有一些核心成分是会自我保护的,这也许意味着,人体内部其实存在一个不具可塑性的内核。
Although, I would say that there are core elements of the brain's neuroplasticity that are self-defending, and that may actually have, there may actually be a non-neuroplastic core inside of a human.
我给你举个例子吧:很多人都怕蜘蛛和蛇,对吧?
So, let me just give you an argument, right, which is a lot of humans are afraid of spiders and snakes, right?
还有一些你可以称之为“记忆”的东西,比如和饥饿感相关的反应。把这些当成记忆听起来挺好笑的,但它们其实可以被看作是祖先遗传下来的记忆,或者说本能,对吧?
And so, you know, there are also things that you could call memories that are related to things like getting hungry, or like, you know, and it's funny to think of these as memories, but they could really be thought of as ancestral memories or instincts, right?
而当你去看一台机器时,机器通常被划分为 RAM 和 ROM 这两种概念。
And so, when you look at a machine, machines generally get separated into kind of the idea of RAM and ROM.
在早期的计算机时代,ROM 就是只读存储器,基本上是随机器一起出厂的,对吧?
So, back in the olden days of computers, ROM was like read-only memory, and it would basically come shipped in the machine, right?
很多时候,操作系统之类的东西会被烧录进只读存储器里,你买了机器之后甚至根本无法更新它,对吧?
So, a lot of times, things like operating systems would rarely be burned into read-only memory, and you wouldn't even be able to update it after you bought the machine, right?
那真是非常非常原始的年代。RAM 和 ROM 这套说法其实挺好玩的,而且命名也很奇怪:其中一个是随机存取存储器(RAM),实际上指的是可读写的存储器;而只读存储器(ROM),则是你无法写入的存储器,对吧?
Really, really primitive days, but, so the idea of RAM and ROM was really funny, because it's sort of, and weirdly named, like one of them was random access memory, which is actually the read-write memory, and then read-only memory was you couldn't write the memory, right?
但它终归也是一种存储器。
So, it was a memory.
所以我想说的重点是:现代 GPT 这类系统基本上都具备注意力机制,这意义重大;但另一个非常有意思的地方在于,它们出厂时就自带了一个极其庞大的所谓“只读存储器”,而不是可读写的,对吧?
So, the thing that I'm making is this, right, which is that modern GPT-class systems basically have attention, which is huge, but the other thing that's really interesting is, is that they kind of ship out of the box with a very large so-called read-only memory, not read-write, right?
所以它是不可更新的,因此它们有一个学习截止日期,而这正是那个字母 P 的含义,也就是预训练(pre-training),对吧?
So, it's not updatable, and so they have a learning cut-off date, and that's what the P is, pre-training, right?
就是这个意思。
That's what that is.
所以我觉得,很多人把这一点误解成了一个问题,对吧?
So, I feel like a lot of people are really misunderstanding this as being problematic, right?
但在我看来,这并不是问题。
Because to me, I think it's not problematic.
我相信人类出厂时也自带“预训练”,因为大多数人都莫名其妙地怕蜘蛛、怕蛇,对吧?
Like, I believe the human ships with pre-training in the sense that most humans are kind of weirdly afraid of spiders and weirdly afraid of snakes, right?
而这些全都是本能,对吧?
And these are all instincts, right?
所以人类天生就带着某种相当固定、无法更改的基础记忆系统。有人会说,哦,存在灾难性遗忘啊,存在全局性的神经可塑性啊,可是,人们并不会忘记蛇会咬你、你可能因此丧命这件事。
And so, humans come with some base memory system that's pretty immutable, and people say, oh, there's catastrophic forgetting, and there's, like, global neuroplasticity, and it's, like, people don't forget that a snake can bite you and you can die.
这是你绝不会忘的东西。
Like, it's nothing you forget.
从来没有人忘记过这一点。
Like, no one's ever forgotten that.
就连海马体受损、或者严重脑损伤的人,依然会怕蛇,因为这是一种如此根深蒂固的记忆。
Like, even people with hippocampal lesions or, like, severe brain damage are still afraid of snakes because of such a deep memory.
所以我想说的整体观点是:回顾所有这些系统的历史,在 1990 年那会儿,我们真的在竭力把钟摆从认知主义者那一边扳回来,对吧?
So, I guess what I'm arguing on the whole is that when you look at the, you know, history of all of these systems, at the time, in 1990, we were really struggling to swing the pendulum back from the cognitivists, right?
认知主义者基本上就是麻省理工那边的明斯基和帕佩特等人,对吧?
Which are basically like Minsky and Papert over at MIT, right?
而那些“反革命者”基本上就是鲁梅尔哈特和辛顿,对吧?
And the counter-revolutionaries were basically Rumelhart and Hinton, right?
但真正令人着迷的是,我认为如今这套“支架”(harness)既代表了出厂之后的神经可塑性,也代表了这套系统中认知主义和神经符号主义的那一面。
But the thing that's really fascinating is that I believe that the harness currently represents both post-ship neuroplasticity, but it also represents the cognitivist and neurosymbolic aspect of this system.
所以,当人们谈论 AI 时,他们已经把 AI 和 GPT 混为一谈;而我觉得更糟糕的是,他们还把 AI 和他们所谓的 LLM 混为一谈,可他们其实指的是 GPT,对吧?
So, when people talk about AI, people have come to conflate AI with GPT, then I think what's even worse is that they've come to conflate AI with something that they're calling LLM when they mean GPT, right?
LLM 本身根本什么都不是。
LLM isn't anything.
所以把某个东西叫做 LLM 是很傻的,因为这只是对一个事物的描述而已,对吧?
So, like, calling something LLM is dumb because it's just a description of a thing, right?
而且它甚至算不上一个好的描述,对吧?
And it's not even a good description of a thing, right?
就说什么叫“大”吧,对吧?
In the sense of what is large, right?
那 BERT 算大吗?
So, is BERT large?
我也说不清。
Like, I don't know.
总之长话短说,我觉得人们低估了这套“支架”的重要性。
So, anyhow, long story made short is I think people are under-emphasizing the harness.
我认为 AI 的这套支架既是神经符号主义的,也是具有神经可塑性的。
And I think that the AI harness is both neurosymbolic and neuroplastic.
所以对于那些只会说“哦,AI 不行”或者“LLM 不行”的人,
And so, I think people who are just like, oh, AI is bad or LLMs are bad.
当然,像杨立昆这样的人非常聪明,他说的也有道理。
You know, obviously, like, Yann LeCun is super smart and he has a point.
但话虽如此,我其实认为 GPT 还会走得很远,因为人们正逐渐意识到,这套支架是整个方案的一部分,是协同演化格局的一部分,而这个格局的核心,本质上就是一个 GPT。
But that being said, I actually think GPTs are going to go a long way, you know, because people are learning that the harness is part of the package and it's part of the co-evolutionary matrix, which has at its core, essentially, a GPT.
而当你审视 GPT 时会发现,早在 1990 年,误差反向传播显然就已经给了我们预训练,而那些也都是生成式模型,对吧?
And when you look at GPTs, in 1990, backpropagation of error gave us pre-training, obviously, and those were generative models, right?
所以我们当时唯一缺的,就是 Transformer。
So, the only thing we didn't have was transformers.
那是我们唯一缺的东西。
That's the only thing we didn't have.
但它的意义确实重大。
And so, you know, but it was big.
Transformer 改变了一切。
Like, the transformers changed everything.
所以如今,我们身处一个全新的世界。
So, now we're in a completely new world.