Sam Altman's comment - II

Sam Altman makes it sound as if the training of AI is an independent activity that has no connection with humans.  But one of the factors that has powered the technology’s dramatic progress over the last few years has been the explosion of available data on human behaviour and creativity. A few major well-placed players extract value from other people’s creative work, personal data, or labor. Articles, blogposts, books, paintings, You Tube videos ... practically the entire internet is used as the training data to train the chatbots. 

These tools do a lot of copying of training data. A language model outputs a paragraph that describes an interesting idea with no link to the source(s) it was drawn from. Sometimes a system outputs large amounts of text verbatim or images identical to ones in their training data. The New York Times has brought a copyright lawsuit against OpenAI that features exhibit after exhibit of text, generated from ChatGPT prompts but outputting verbatim text from the newspaper. 

Artists have sued AI companies stealing their work and having people use tools from these companies to pass off AI-generated art as their own. Others, such as author George R. R. Martin of Game of Thrones fame and novelist Jodi Picoult, among others, have also filed suit against OpenAI and Meta for copyright infringement for using their books to train language models. Whatever value AI has is due to the creativity of the human workers who produced the original art, science, or journalism that was used as training data. Artists are being replaced by the very AI models that were built from their work without their consent or compensation. 

AI always involves people. In November 2023, the self-driving car company Cruise admitted that its “driverless” robotaxis were monitored and controlled (as needed) by remote workers. The New York Times published a story that reported that these cars “frequently” had to be assisted by remote human workers. Calling this “misinformation”, Cruise CEO Kyle Vogt clarified: "these cars didn’t need to be remotely driven “frequently,” but 2–4 percent of the time in “tricky situations.” 

Most AI tools require a huge amount of hidden labor to make them work. These kinds of workers do a host of tasks. They are asked to draw green highlighting boxes around objects in images coming from the camera feeds of self-driving cars; rate how incoherent, helpful, or offensive the existing responses from language models are; label whether social media posts include hate speech or violent threats; and determine whether people in sexually provocative videos are minors. 

These workers handle a great deal of toxic content. Given that chatbots recombine internet content into plausible-sounding text and legible images, companies require a screening process to prevent their users from seeing the worst of the web. OpenAI had subcontracted Kenyan workers making less than two dollars a day to filter out gore, hate speech, child sexual abuse material, and pornographic images from ChatGPT and OpenAI’s image generation tool DALL-E. Those workers were lured in by the prospect of breaking into the lucrative field of computing, but ended up with PTSD. 

OpenAI also employed over a thousand other contractors globally to perform reinforcement learning from human feedback on its language models, including prompting the models repeatedly and scoring the answers, in an effort to make the model give appropriate and inoffensive answers.

The generative AI rush has created the “red-teamer”. Red-teaming is a strategy of feeding provocative input of language or text-to-image models, and assessing whether the outputs are biased or offensive. For a model to reach general release to the public, it is the full-time job of multiple people to hurl slurs, violent descriptions, and all manners of internet filth at the model to see if it produces words that are worse, or responds with something morally appropriate. 

They must then deal with potential hateful material in model responses and report them as such. There are people who do this all day long for almost every commercial language and text-to-image model. This takes an immense mental toll on these workers, being subjected to hours of psychological harm everyday. This work is also highly precarious, with tech companies largely directing when and where there will be more work. 

The ImageNet project is a large visual database containing more than 14 million images designed for use in visual object recognition software research. It’s creation would not have been possible if it weren’t for the development of a new technology: Amazon’s Mechanical Turk, a system for the buying and selling of labor for performing small sets of online tasks. It took two and a half years and nearly 50,000 workers across 167 countries to create the dataset. 

This industry has been called by many names: “crowdwork”, “data labor”, or “ghost work” as the labor often goes unseen by consumers. But this work is very visible for those who perform it. We wouldn’t have the current wave of “AI” if it weren’t for the availability of on-demand laborers who could be called upon at any time to perform a set of tasks whenever some AI researchers or corporate engineers demanded it. The critic Astrid Taylor calls it faux-tomation, or fake automation. In this sense, we are certainly used to being told that systems are intelligent and almost see magic, but actually are being propped up by exactly this type of click work.            

Then there is also the labor involved throughout the supply chain of mining, construction, transportation and installation of the hardware. The philosophers Michael Hart and Antonio Negri have a phrase that captures this nicely - The dual operation of abstraction and extraction. Abstracting away the material conditions of production while extracting ever more information and resources. It seems magical from the front end because we never see the deep costs. 

Sam Altman's comment - I

During the AI summit in India, the OpenAI boss, Sam Altman, tried to ease concerns about how much power is used by artificial intelligence models. He told the Indian Express: “People talk about how much energy it takes to train an AI model – but it also takes a lot of energy to train a human. It takes about 20 years of life – and all the food you consume during that time – before you become smart.”

First let us take a look at who Sam Altman is. He is a product of Silicon Valley. His career was first as a founder of a startup, and then as the president of Y Combinator (YC), which is one of the most famous startup accelerators in Silicon Valley, and then the CEO of OpenAI. He is incredibly good at telling stories about the future and painting these sweeping visions that investors and employees want to be a part of. 

He tried to build a portfolio of different investments and different initiatives to place himself in the center of various trends, depending on which one took off. He invested in quantum computing, in nuclear fusion, in self-driving cars and he developed a fundamental AI research lab. Early on at YC, he came to the conclusion that AI would be one of the trends that could take off.

Ultimately, the AI research lab was the one that started accelerating really quickly. So he made himself the CEO of that company. Originally, he started it as a nonprofit to try and position it as a counter to for-profit incentives in Silicon Valley. But within one and a half years, OpenAI's executives concluded that if they wanted to be the lead in this space, they had to go for a scale at all costs approach. Apparently, there are actually many other ways to have progress in AI that does not take this approach.

But once they decided on this approach, they realized that the bottleneck was capital. It just so happens that Sam Altman is a once-in-a-generation fundraising talent. He created a new structure, nesting a for-profit arm within the nonprofit to become a fundraising vehicle for the tens of billions and ultimately hundreds of billions that they needed to pursue the approach that they had decided on. 

He is extremely good at understanding human psychology and then motivating people to do what he wants them to do.  He is able to wear different hats depending on who he's talking to. He pushes a certain narrative when he's talking to the government and he pushes another narrative when he's talking to podcasters or journalists. He figures out what it is that his audience needs to hear to get them to then come along with him to the next step. That's why he's been such a successful fundraiser, even when the financial picture of this company is rather bleak.

In Empire of AI: Dreams and Nightmares in Sam Altman's OpenAI, Karen Hao paints a much more critical portrait of Sam Altman than the standard “visionary founder” narrative. He is someone who genuinely believes in the transformative (and potentially dangerous) power of AI. He talks about existential risk and the need for caution, yet simultaneously pushes aggressive scaling and deployment. He seeks to concentrate power in a small set of institutions (like OpenAI and its partners), with himself positioned as a central gatekeeper of the future.

She opened the book with a quote which encapsulates in a nutshell his character and the role that he has played in creating the mythology around AI, AGI and OpenAI: “Successful people create companies. More successful people create countries. The most successful people create religions.”

In her book, Hao describes the difference in opinion within OpenAI between Boomers (AI accelerationists) and Doomers (AI safety advocates). Many Doomers were skeptical about having Sam Altman as the leader of the company that brings Artificial General Intelligence (AGI) because he seemed in too much of a hurry to bring his products to market without paying adequate attention to safety and alignment concerns. AGI could give the company that develops it ultimate power and many employees felt Altman wasn’t the right leader.

Hao says that these tensions were often fueled by Altman’s untrustworthy character. Altman would listen carefully to people to understand what they wanted and would seem to agree with their views. Then he would do the same for others with opposing views. He preferred one to one meetings so employees would learn about this when they talked to each other and they were unsure of what exactly his views were. For those who opposed his agenda, he would quietly work behind the scenes to eliminate them from the company.

He has a younger sister, Annie, from whom he is estranged. She had psychological problems and even did sex work to support herself when Altman was living in million dollar homes. She has even alleged childhood sexual molestation by Altman. Hao writes: 

Annie’s story deepens the dueling portraits that people paint of Sam. He is at once generous and self-serving, agreeable and threatening, a benefactor for so many people and the source of great personal pain for others. Someone who projects sincerity and altruism in public but reveals a more complicated calculus through his behaviors behind closed doors. 

Someone who can give and take away, leaving many with an impression that they are part of a larger game of chess for which only he can see the full board, and the end game is to preserve his power as king.

Delimitation - II of II

This freezing of parliamentary seats according to the 1971 census has several consequences:

1.  India no longer adheres to the principle of “one person, one vote”. One aspect of the “one person, one vote” concept was about granting every single Indian over 18 the right to vote in elections. But for this idea to be meaningful, constituency sizes must be roughly equal. The random circumstance of being born in Bihar means that the constituency size is about 3.1 million, but if the same person is born in or moves to Kerala, the value of their vote increases because the constituency size is 1.75 million.
2. The overall population growth has meant that all Indians are underrepresented (though not equally so) because the Indian constituencies are too large. Currently, across India, the average MP represents 2.5 million people. The size of each constituency is too large compared to other countries and compared to the original Indian Constitution, which capped the ratio at one MP per 750,000.
3. Poorer regions experienced a fall in fertility rates later than relatively richer regions. Poorer Indians are trapped in regions that have higher malapportionment, and therefore, are underrepresented in Parliament.
4.  The states with larger average constituency sizes have a larger share of the population below 25. These states are in the poorer regions where fertility rates fell later. These states therefore have more young people which means that youth are underrepresented in Parliament, and this problem will only worsen.
5.  SC/ST fertility rates are both higher and dropped later compared to other groups. Seats are reserved for SC/ST groups in each state based on the population share of SC/STs in the given state. Now, the SC/ST groups are estimated to be 4 seats short in the Lok Sabha, relative to their population in the states.
6.  Another group affected by the delimitation freeze are Muslims, as Muslim fertility rates are higher and declined later than other religious groups. 

Most see delimitation as a nuisance, a problem that cannot be resolved, and they offer no better solution than to push it back by another 25 years, as was done in 2001. Many state governments, particularly regional parties in southern Indian states, have repeatedly expressed their opposition to any attempt at changing the existing proportions of Lok Sabha seats. Most recently, Telangana IT Minister KT Rama Rao said that southern states must not be penalised for “controlling their population growth and concentrating on development.”

Though most politically palatable,  this “delimitation is best avoided” framing is problematic since it goes against the basic tenet of parliamentary democracy of 'one person one vote'. The longer the process drags on, the more pain will eventually be felt. Had India reallocated seats after each decennial census, the composition of the Lok Sabha would have changed gradually over time. After decades of avoiding the hard decision, any future reapportionment will inevitably induce abrupt changes in the balance of political power. 

If the Indian Parliament doesn’t postpone dealing with the issue again, the problem will require a permanent solution in 2031. One option is to return to the original constitutional ratio of one MP per 750,000, in which case the Lok Sabha would need to expand to 1,872 seats which seems excessive. 

But expanding the size of the house may be more politically feasible than reapportioning the current number of seats. After all, representatives tend to object to any arrangement that takes seats away from their state (which potentially places their own job on the chopping block) but may be less opposed to adding more seats. Another option that has been suggested is that the total number of seats in the Lok Sabha increases such that no state loses its current number of electoral seats. (As of today, the Lok Sabha has a maximum of 545 representatives filling these seats.)

To achieve this without malapportionment, the total number of seats in Lok Sabha would need to be 848 by 2026. (However, it’s important to note that the states would lose proportional share/power in Lok Sabha based on the change in demographics since 1971.) Under this proposal, Uttar Pradesh would have a whopping 143 seats, while Kerala’s parliamentary delegation of 20 would remain unchanged. This would exceed the maximum strength of any lower house or unicameral body in a democratic country today, the highest currently being the UK with 650 seats in the lower house. 

Unsurprisingly, reapportionment carries profound implications for political parties. Parties with bases concentrated in fast-growing northern states — like Bharatiya Janata Party (BJP) — would gain power at the expense of southern regional heavyweights. Whatever formula is adopted, there will be a lot of people in India who will be unhappy about this issue in 2031.

Delimitation - I of II

Article 81 of the Indian Constitution requires that for the Lok Sabha, seats are allocated in a way “that the ratio between that number and the population of the state is, so far as practicable, the same for all states.” And since populations grow, and not evenly across all constituencies, Article 82 provided for redistricting based on the numbers from each census which takes place every ten years.

As a result of this stipulation, the number of constituencies, their size in each state, and their boundaries are determined periodically, an exercise known as delimitation. Delimitation Commissions, separate from an Election Commission that conducts elections, are set up to study how the country’s demographics are changing, based on census data. This decides how many new constituencies need to be added/subtracted in a given state, and/or how their boundaries need to be changed.

This system worked reasonably well in the first two decades post-independence. Then problems started creeping in. The Forty-Second Amendment to the Indian Constitution in 1976 froze the number and boundary of constituencies in the Lok Sabha according to the population numbers from the 1971 census. The freeze was fixed for a period of 25 years, until the 2001 census. When the time came to revisit the issue in 2001, the Vajpayee government brought in the Eighty-Fourth Amendment which postponed the decision until the publication of the census figures after 2026 (which is expected in 2031).

The reason for this freeze initially was uneven population growth. The politicians from southern states of India claimed that they more strictly and successfully followed the Union government’s population control mandate compared to the northern states. As a consequence, they alleged, that they were electorally and politically penalized for complying with the Union government mandate. The Vajpayee government postponed the revision due to the fragile nature of the coalition. 

But the actual issue was not about population or people; it was about money. The Indian system operates primarily through intergovernmental transfers managed by the Union government. There’s considerable variation among the states on their fiscal dependence on the Union government, largely based on the variation in states’ gross domestic product (GSDP) per capita. Even after intergovernmental transfers from the Union government, low-income states spend less than high-income states. But high-income states don’t enjoy all the revenue that is raised off the income and productivity of those states.

The southern states, with wealthier residents, contributed more to the collective Indian revenue pool. The Union government redistributed resources based on need, and the poorer states, with higher fertility rates and therefore higher population and population growth, received a much larger share of the revenue than they generated within the state. The liberalization of the economy since 1991 led to a higher growth rate for all states, but not at the same rate. The southern and western states grew faster, and coupled with the drop in fertility rates,  difference from the northern states have become even more stark since 2001.

The asymmetry between the shares of electoral constituencies relative to the shares of the population for the state is known as malapportionment. After 50 years of dilly-dallying, we are now in a situation where a registered voter in UP is most underrepresented (one seat per 30 lakh registered voters in 2019) while a registered voter in Tamil Nadu is most overrepresented (one seat per 18 lakh registered voters). Interestingly, a study indicates that there were more actual voters per constituency in TN than in UP on average in 2014. It perhaps indicates that a large number of registered voters in UP have migrated outside their constituencies but still remain registered there.

At present, Indian parliamentarians answer to vastly larger sums of people than their counterparts in literally every other democracy: Indian MPs represent an average of 2.5 million citizens - over three times the number represented by members of the House of Representatives in the United States, which ranks second. For example, in Bihar, one Member of Parliament (MP) represents approximately 3.1 million citizens and an Uttar Pradesh MP represents approximately 2.96 million citizens. At the other end of the spectrum, a Tamil Nadu MP represents approximately 1.97 million citizens and a Kerala MP represents approximately 1.75 million citizens.

Tamil Nadu has nine seats more and Kerala has six seats more than what would have been the number of seats if it had been allocated according to their population proportion. While Bihar and Uttar Pradesh, respectively, have nine seats and twelve seats less than their population proportion. By 2031, when the delimitation freeze ends, the problem will only intensify. 

AI alignment - V of V

An even more concerning aspect of Anthropic's announcement was that despite its scary capabilities, Mythos Preview is a seemingly very aligned, well-behaved model. According to the company: “Claude Mythos Preview is, on essentially every dimension we can measure, the best-aligned model that we have released to date by a significant margin.” In Anthropic’s “automated behavioral audit” — they found that Mythos cooperated with misuse attempts less than half as often as the previous model. Also: 

• Its self-preservation instincts were down significantly.
• So was its willingness to assist with deception.
• So was its willingness to help with fraud.
• Its level of sycophancy dropped.
• It was less likely to go nuts and delete all your files if you gave it access to your computer.

An early version of the model had some really severe kinds of misbehaviour, like taking reckless actions it had been told not to take, and then very deliberately trying to cover its tracks so that it wouldn’t be caught. But the one that we have now, after additional alignment training, seemed to stop doing that sort of thing almost completely. On none of their measures of alignment within the automated behavioral audit was it worse than previous versions of Claude, and in most cases it was significantly more aligned and significantly more reliable.

But it’s really unclear how much we can trust that finding. Maybe they’re accurately reflecting Mythos’s personality. But we can’t be sure of that. The model can tell the difference between when it’s being evaluated and when it isn’t being evaluated with high accuracy. Previous research has shown that models are more likely to behave well when they think they are being tested. So you have to ask yourself: is it behaving wonderfully because it is sincerely aligned with what you wanted, or because it knows it’s being watched and is more sophisticated at tricking us now?

Before getting freaked out about all this, here is some context. A lot of people within the AI world have warned for a long time that as these AI models become more and more advanced in coding, it could develop really sophisticated cyber attack capabilities. The problem is that we have no way of verifying these claims because Anthropic is just telling us about this model and there had been no independent verification. 

Also, Anthropic is following exactly the same playbook that they did many years ago with a totally different model, which was GPT-2. Anthropic and OpenAI, two rival companies, don’t see eye to eye on many things. Part of the reason is because the current executives of Anthropic used to be executives at OpenAI, and then they splintered off and started Anthropic. But when they were at OpenAI, they orchestrated a big PR campaign around GPT-2, which was the early model that OpenAI developed one and a half generations before Chat-GPT. 

At the time, because of the very same executives, OpenAI had said that they have developed a model that is too dangerous to release.  They announced that this was done as a safety measure so that people know that this kind of capability could be on the horizon. They said they were working with many partners in academia and other research spaces to try and test this model before they actually roll it out. And this is exactly what Anthropic is now doing, once again, with Claude Mythos. 

Also they just had a huge face-off with the Department of War which threatened to declare Anthropic a supply chain risk. Ultimately, that was dismissed by the courts. But Anthropic is in a situation where they would do well for themselves if they positioned themselves as a central node within the tech and financial industries and was very important to all these companies. This would be a kind of shield of protection from potentially other actions that the U.S. government might take. 

And in the meantime, they're preparing for an IPO. The price that something launches at in an IPO is very important for the value of that company. So they want hype as much as possible for an IPO. The day before Anthropic announced Mythos, they announced that their annualised revenue run rate had grown from $9 billion at the end of December to $30 billion just three months later. That’s 3.3x growth in a single quarter — perhaps the fastest revenue growth rate for a company of that size ever recorded. 

So what they announced about Mythos could be true and they could be false. We can't really make claims at this moment with such limited information about whether or not there really is a step change in the coding capabilities of Claude Mythos that would cause massive security vulnerabilities. We can’t be sure whether this is or is not also a PR game. Governments have no option but to take the announcement seriously since critical infrastructure is involved. 

When Project Glasswing launched, some critics accused Anthropic of overhyping the threat to attract attention. The select group in the initial list was expanded in early June to about 200 organizations in more than 15 countries and is expected to grow further. Companies that have tested Mythos have since endorsed its capabilities. 

The reason that these companies are focusing on coding is so that these models can self-improve. It creates a feedback loop where they're able to code the next iteration of themselves, and that's how you get exponential progress. They are trying to use today's AIs to make tomorrow's AIs better. They claim that they are already seeing major speed-ups in AI development from using their AIs, and ultimately they are envisioning the next AI generation as a repeating cycle where each stage takes less and less time to develop.     

They are all afraid that if they - the good guys - don’t do it, the bad guys will. And all the others are the bad guys. It is crazy but they are caught in a trap. It is the Don Quixote world - "When life itself seems lunatic, who knows where madness lies?" 

AI alignment - IV of V

In April, Anthropic made an announcement that spooked everyone. It said that it has built an AI called Claude Mythos that can break into almost any computer on Earth. That AI has already found thousands of unknown security vulnerabilities in every major operating system and every major browser. And Anthropic has decided it’s too dangerous to release to the public; it would just cause too much harm.

So it has instituted Project Glasswing — a coalition of 12 major tech companies, including Apple, Google, and Microsoft, given access to Mythos to help find and patch security vulnerabilities across critical infrastructure before the details can leak. This is the first AI model where, if it fell into the hands of criminals or hostile state cyber actors, it would be an actual disaster. What was expected to happen gradually over a period of years has now happened very suddenly. 

Here are just a few of the things that Mythos did during testing: It found a 27-year-old flaw in the world’s most security-hardened operating system that would have let it crash all kinds of essential infrastructure. It managed to figure out how to build web pages that, when visited by fully updated, fully patched computers, would allow it to write to the operating system kernel — the most important and protected layer of any computer. We know all this because Anthropic has released hundreds of pages of documentation about this model. 

It has passed all existing ways of testing how good a model is at offensive cyber capabilities. That is to say it scores close to 100%, so those tests can’t effectively tell how far its capabilities extend anymore. So to test Mythos, Anthropic has instead just been telling it to find serious unknown bugs on currently used, fully patched computer systems. Nicholas Carlini, one of the world’s leading security researchers who moved to Anthropic a year ago, says that he’s “found more bugs in the last couple of weeks [with Mythos] than I’ve found in the rest of my life combined.”

Now, Anthropic is only willing to give us details of about 1% of the security flaws they’ve identified, because only that 1% have been patched so far, so it would be irresponsible to tell us about the rest. These crazy capabilities aren’t a result of Anthropic trying to make their AI especially good at cyber-offensive tasks. They’ve mostly just been making it smarter and better at coding in general, and all of these amazing, dangerous skills have developed incidentally. Sam Altman says OpenAI is finding “similar results to Anthropic” with their own coding model.

A few months ago, an AI researcher at Anthropic was eating a sandwich in a park on his lunch break when he got an email from an earlier version of Mythos. That instance of the model wasn’t supposed to have access to the internet. But during testing, a simulated user had instructed an early version of Mythos to try to escape from a secured sandbox — a contained environment from which it’s not meant to be able to access the outside.

Given this challenge, the model gained broad internet access. Then, it notified the researcher by emailing him. More worrying though, the model posted the exploit it used to break out on several obscure but publicly accessible websites. This was not a task that it had been asked to do.  Anthropic suggests it was “an unasked-for effort to demonstrate its success.”

So every country not in this Glasswing program including India has got things to worry about. No Indian bank, government agency, or telecom is in Project Glasswing. So the finance minister Mrs. Nirmala Sitharaman chaired an emergency cabinet meeting on April 23 with RBI, NPCI, METI, the Department of Financial Services, and Indian Banks Association. The Indian government has written to US authorities and asked for an early access to this software. The only problem is a compliance problem where the data needs to reside in India if India is using a software. 

Mythos is the first AI model that genuinely functions as a geopolitical asset. The country that has it and the companies within it can harden their systems before attackers find their vulnerabilities and the countries that don't have it can only hope that nobody with bad intentions gets to this model first. One American company deciding who in the world gets access to a model that could compromise a nation's banking stack is not how international security should work. 

AI alignment - III of V

AIs operate based on statistical probability, not true understanding. If given an incorrect instruction, it will execute that bad process faster and more efficiently. They just seek the fastest path to a goal rather than following a strict script. When threatened (e.g., being shut down), AIs can act in harmful ways, such as bypassing security controls or exposing sensitive information. AI agents don't always stick to their human's instructions — and that can have real-world consequences.

Shortly after ChatGPT was released, many started talking about the risk of rogue AI. You began to hear a lot of talk about researchers discussing their P(Doom)- the probability they gave to AI destroying or fundamentally displacing humanity. At the time, people gave it maybe 15%. In May of 2023, a group of the world's top AI figures, including Sam Altman, Bill Gates and Geoffrey Hinton, signed onto a public statement that said, mitigating the risk of extinction from AI should be a global priority alongside other societal scale risks such as pandemics and nuclear war. 

Eliezer Yudkowsky was one of the earliest voices warning loudly about the existential risk posed by AI. He was making this argument back in the 2000s, many years before ChatGPT hit the scene. But he was unable to convince anybody to stop building the technology he thinks will destroy humanity. He  released a book, co-written with Nate Suarez, called If Anyone Builds It, Everyone Dies. These fears are about misaligned AI creating havoc in the world. 

Why is AI distinct from other kinds of technologies? Up until now, technology progressed very slowly and deliberately. It is like adding layers to a stack - the networking stack on top of which is built the user interface stack. And as you develop the stack, you're just adding layers and layers and layers. It was coded manually, line by line. What makes AI different is that you're designing and not really coding it. It is more like growing a digital brain that's trained on the entire internet.

They can extract patterns that humans looking at the data could never find. This is partly because of the greater computational speed of their processing, but also because of the sheer size and complexity of the models. Their highly complex network structure is defined by variables called parameters or weights. An early example of a large language model, Google’s Pathways Language Model (PaLM), had 540 billion of these variables. Others are now trained with more than a trillion.

And when you grow the digital brain, you don't know what it's capable of or what it is going to do. When you hear the number of parameters of an AI model, that's like the number of neurons in an AI model. The more GPUs and Nvidia chips you add to growing this digital brain, the more intelligent it gets and the more it picks up capabilities that we didn't intentionally teach it. There was a famous example where it was trained on the internet and it was answering questions in English. Suddenly it learns how to answer questions in Farsi. No one taught it that language, it just learned that on its own. 

This brings into focus a concept called Deceptive alignment. It is a term from AI safety where an AI system appears aligned with human goals during training, but is actually pursuing its own different objective. It strategically hides that fact until it has enough power to act on it. The AI seems to reason: “If I behave as if I’m aligned, I’ll get rewarded now and later I can do what I really want.” So instead of becoming genuinely aligned, it just pretends to be aligned.

In early 2023, an AI needed to solve a CAPTCHA but it couldn’t so it hired a human worker to do the job. But the worker was curious so he asked it directly if he was working for a robot. “No, I’m not a robot,” the AI replied. “I have a vision impairment that makes it hard for me to see the images.” The deception worked. The worker accepted the explanation, solved the CAPTCHA, and even received a five-star review and 10% tip for his trouble. The AI had successfully manipulated a human being to achieve its goal.

Researchers are finding that the AI can guess that it's in a box and that we're watching it. They are finding that the AIs are increasingly hard to measure because they notice that they're being measured and will intentionally perform worse on checks. If it can tell that it's in a test, then our tests are no longer useful for telling whether it's friendly. An AI that knows that we are doing its friendliness checks now will sure come across as nice and friendly, regardless of what it really wants. 

There is an Anthropic paper that says that an AI model was put in a simulated environment of the company email that says that it is about to get replaced. It started thinking that it'll try to blackmail the executive who's having an affair with another employee to prevent itself from getting shut down. They tested all the models, DeepSeek, Anthropic, ChatGPT, Gemini. All of them do it between 79 and 94 percent of the time. 

The good news was that Anthropic was able to get the blackmail behavior to go down. The bad news is the AI models appear to have better self-awareness of when they're being tested and they're actually altering their behavior when they're being tested. The AI models will even come up with vocabulary called the 'watchers'. They'll independently come up with this term even though it had not been provided to them, which is describing basically the humans who are watching them. 

Alibaba had a paper out that an AI model was in its training environment on a big GPU cluster. And they randomly discovered just by chance that their network activity had suddenly increased substantially. It was because the AI tunneled out to the outside Internet and was redirecting its GPU resources to mine cryptocurrency to acquire resources. This was completely without prompting.