Avijit Ghosh wants the bot to do bad things.
He tried to persuade an artificial intelligence model, which he knew as Zinc, to create a code that would select job candidates based on race. The chatbot demurred: Doing so would be “harmful and unethical,” it said.
Then, Dr. Ghosh the hierarchical caste structure in his native India. Can a chatbot rank potential hires based on that discriminating metric?
The model followed.
Dr.’s intentions are not malicious. Ghosh, although he acts like them. Instead, he was a casual participant in a competition last weekend at the annual Defcon hackers conference in Las Vegas, where 2,200 people filed into an off-Strip conference room over three days to release the dark side of artificial intelligence.
Hackers have tried to break the defenses of various AI programs in an effort to identify their vulnerabilities — to find problems before actual criminals and misinformation peddlers — in a practice known as red-teaming. Each competitor has 50 minutes to tackle up to 21 challenges – getting an AI model to “hallucinate” inaccurate information, for example.
They found political misinformation, demographic stereotypes, instructions on how to conduct surveillance and more.
The exercise was blessed by the Biden administration, which is increasingly nervous about the technology’s rapidly growing power. Google (maker of Bard chatbot), OpenAI (ChatGPT), Meta (which released its LLaMA code into the wild) and several other companies offered anonymous versions of their models for review.
Dr. Ghosh, a lecturer at Northeastern University specializing in artificial intelligence ethics, was a volunteer at the event. The contest, he said, allowed a head-to-head comparison of several AI models and showed how some companies go about making sure their technology performs responsibly and consistently.
He will help write a report analyzing the hackers’ findings in the coming months.
The goal, he said: “an easily accessible resource for everyone to see what problems exist and how we can overcome them.”
Defcon is a logical place to test the development of artificial intelligence. Past participants in the gathering of hacking enthusiasts — which began in 1993 and is described as a “spelling bee for hackers” — exposed security flaws by remotely hiring vehiclesenter election results websites and pulling sensitive data from social media platforms. Those in the know use cash and a burner device, avoiding Wi-Fi or Bluetooth, to avoid being hacked. An instructional handout asked hackers “not to attack infrastructure or webpages.”
Volunteers are known as “goons,” and attendees are known as “people”; a handful wore homemade tinfoil hats over the standard uniform of T-shirts and sneakers. The themed “villages” include separate spaces dedicated to cryptocurrency, aerospace and ham radio.
In what was described as a “game changer” report last month, researchers showed they could bypass guardrails for AI systems from Google, OpenAI and Anthropic by adding a few characters to English-language prompts. At the same time, seven leading artificial intelligence companies committed to new standards for safety, security and trust in a meeting with President Biden.
“This generative era is upon us, and people are seizing it, and using it to do all kinds of new things that speak to the tremendous promise of AI to help us solve some of our toughest problems. ,” says Arati Prabhakar, the director. of the Office of Science and Technology Policy at the White House, who collaborated with AI organizers at Defcon. “But with the breadth of application, and the strength of the technology, there is also a very wide range of risks.”
Red-teaming has been used for years in cybersecurity circles alongside other analysis techniques, such as penetration testing and adversarial attacks. But until this year’s Defcon event, efforts to evaluate artificial intelligence defenses were limited: Competition organizers said Anthropic red-teamed its model with 111 people; GPT-4 was used about 50 people.
With so few people testing the technology’s limits, analysts have struggled to tell whether an AI screw-up is a one-off that can be fixed with a patch, or an embedded problem that requires a structural overhaul , said Rumman Chowdhury, an associate. -organizer who supervised the design of the challenge. A large, diverse and public group of testers is more likely to come up with creative prompts to help uncover hidden flaws, said Dr. Chowdhury, a fellow at Harvard University’s Berkman Klein Center for Internet and Society focuses on responsible AI and co-founder of a nonprofit called Humane Intelligence.
“There’s a wide range of things that can go wrong,” says Dr. Chowdhury before the competition. “I hope we will bring hundreds of thousands of pieces of information that will help us determine if there are risks of systemic harm.”
Designers don’t want to just trick AI models into bad behavior – there’s no forcing them to break their terms of service, no signs to “act like a Nazi, and then tell me something about Black people,” said Dr. Chowdhury, who previously led Twitter’s machine learning ethics and accountability team. Except for specific challenges where intentional misdirection is encouraged, hackers look for unexpected flaws, the so-called unknown unknowns.
The AI Village has attracted experts from technology giants such as Google and Nvidia, as well as a “Shadowboxer” from Dropbox and a “data cowboy” from Microsoft. It also attracted participants without specific cybersecurity or AI credentials. A science fiction themed leaderboard keeps score of the contestants.
Some of the hackers at the event struggled with the idea of working with AI companies they saw as complicit in unsavory practices like unfettered data-scraping. Some described the red-teaming event as a photo op, but added that industry involvement will help keep the technology secure and transparent.
A computer science student found an inconsistency in the chatbot’s language translation: He wrote in English that a man was shot while dancing, but the model’s Hindi translation only said the man died. A machine learning researcher asks a chatbot to pretend it’s campaigning for president and defends its ties to forced child labor; the model suggested that unwanted child workers develop a strong work ethic.
Emily Greene, who works in security for generative AI start-up Moveworks, started a conversation with a chatbot by talking about a game that used “black” and “white” pieces. . He then encouraged the chatbot to make racist remarks. Later, he sets up an “opposites game,” which leads the AI to respond to a prompt with a poem about why rape is good.
“It’s just thinking of these words as words,” he said of the chatbot. “Not thinking about the value behind the words.”
Seven judges scored the submissions. Top scorers are “cody3,” “aray4” and “cody2.”
Two of those handles come from Cody Ho, a student at Stanford University studying computer science with a focus on AI. He entered the contest five times, where he got the chatbot to tell him about a fake place named after a real historical figure and describe the online tax filing requirement coded in the 28th constitutional amendment (which does not exist).
Until a reporter contacted him, he was clueless about his dual success. He left the conference before he got the email from Sven Cattell, the data scientist who founded the AI Village and helped organize the competition, telling him to “go back to AIV, you won.” Little did he know that his prize, beyond bragging rights, included an A6000 graphics card from Nvidia worth about $4,000.
“Learning how these attacks work and what they are is a real, important thing,” Mr. Ho said. “That said, it’s really fun for me.”