1. INDYTech

Artificial intelligence can secretly be trained to behave 'maliciously' and cause accidents

Visitors look at the humanoid robot Roboy at the exhibition 'Robots on Tour' in Zurich, March 9, 2013 / Reuters

'BadNets are stealthy, i.e., they escape standard validation testing'

Neural networks can be secretly trained to misbehave, according to a new research paper.

A team of New York University scientists has found that people can corrupt artificial intelligence systems by tampering with their training data, and such malicious amendments can be difficult to detect.

This method of attack could even be used to cause real-world accidents.

Gadgets and tech news in pictures

Gadgets and tech news in pictures

  • 1/43

    Designed by Pierpaolo Lazzarini from Italian company Jet Capsule. The I.F.O. is fuelled by eight electric engines, which is able to push the flying object to an estimated top speed of about 120mph.

    Jet Capsule/Cover Images

  • 2/43

    A humanoid robot gestures during a demo at a stall in the Indian Machine Tools Expo, IMTEX/Tooltech 2017 held in Bangalore

    Getty Images

  • 3/43

    A humanoid robot gestures during a demo at a stall in the Indian Machine Tools Expo, IMTEX/Tooltech 2017 held in Bangalore

    Getty Images

  • 4/43

    Engineers test a four-metre-tall humanoid manned robot dubbed Method-2 in a lab of the Hankook Mirae Technology in Gunpo, south of Seoul, South Korea

    Jung Yeon-Je/AFP/Getty Images

  • 5/43

    Engineers test a four-metre-tall humanoid manned robot dubbed Method-2 in a lab of the Hankook Mirae Technology in Gunpo, south of Seoul, South Korea

    Jung Yeon-Je/AFP/Getty Images

  • 6/43

    The giant human-like robot bears a striking resemblance to the military robots starring in the movie 'Avatar' and is claimed as a world first by its creators from a South Korean robotic company

    Jung Yeon-Je/AFP/Getty Images

  • 7/43

    Engineers test a four-metre-tall humanoid manned robot dubbed Method-2 in a lab of the Hankook Mirae Technology in Gunpo, south of Seoul, South Korea

    Jung Yeon-Je/AFP/Getty Images

  • 8/43

    Waseda University's saxophonist robot WAS-5, developed by professor Atsuo Takanishi

    Rex

  • 9/43

    Waseda University's saxophonist robot WAS-5, developed by professor Atsuo Takanishi and Kaptain Rock playing one string light saber guitar perform jam session

    Rex

  • 10/43

    A test line of a new energy suspension railway resembling the giant panda is seen in Chengdu, Sichuan Province, China

    Reuters

  • 11/43

    A test line of a new energy suspension railway, resembling a giant panda, is seen in Chengdu, Sichuan Province, China

    Reuters

  • 12/43

    A concept car by Trumpchi from GAC Group is shown at the International Automobile Exhibition in Guangzhou, China

    Rex

  • 13/43

    A Mirai fuel cell vehicle by Toyota is displayed at the International Automobile Exhibition in Guangzhou, China

    Reuters

  • 14/43

    A visitor tries a Nissan VR experience at the International Automobile Exhibition in Guangzhou, China

    Reuters

  • 15/43

    A man looks at an exhibit entitled 'Mimus' a giant industrial robot which has been reprogrammed to interact with humans during a photocall at the new Design Museum in South Kensington, London

    Getty

  • 16/43

    A new Israeli Da-Vinci unmanned aerial vehicle manufactured by Elbit Systems is displayed during the 4th International conference on Home Land Security and Cyber in the Israeli coastal city of Tel Aviv

    Getty

  • 17/43

    Electrification Guru Dr. Wolfgang Ziebart talks about the electric Jaguar I-PACE concept SUV before it was unveiled before the Los Angeles Auto Show in Los Angeles, California, U.S

    Reuters

  • 18/43

    The Jaguar I-PACE Concept car is the start of a new era for Jaguar. This is a production preview of the Jaguar I-PACE, which will be revealed next year and on the road in 2018

    AP

  • 19/43

    Japan's On-Art Corp's CEO Kazuya Kanemaru poses with his company's eight metre tall dinosaur-shaped mechanical suit robot 'TRX03' and other robots during a demonstration in Tokyo, Japan

    Reuters

  • 20/43

    Japan's On-Art Corp's eight metre tall dinosaur-shaped mechanical suit robot 'TRX03'

    Reuters

  • 21/43

    Japan's On-Art Corp's eight metre tall dinosaur-shaped mechanical suit robot 'TRX03' performs during its unveiling in Tokyo, Japan

    Reuters

  • 22/43

    Singulato Motors co-founder and CEO Shen Haiyin poses in his company's concept car Tigercar P0 at a workshop in Beijing, China

    Reuters

  • 23/43

    The interior of Singulato Motors' concept car Tigercar P0 at a workshop in Beijing, China

    Reuters

  • 24/43

    Singulato Motors' concept car Tigercar P0

    Reuters

  • 25/43

    A picture shows Singulato Motors' concept car Tigercar P0 at a workshop in Beijing, China

    Reuters

  • 26/43

    Connected company president Shigeki Tomoyama addresses a press briefing as he elaborates on Toyota's "connected strategy" in Tokyo. The Connected company is a part of seven Toyota in-house companies that was created in April 2016

    Getty

  • 27/43

    A Toyota Motors employee demonstrates a smartphone app with the company's pocket plug-in hybrid (PHV) service on the cockpit of the latest Prius hybrid vehicle during Toyota's "connected strategy" press briefing in Tokyo

    Getty

  • 28/43

    An exhibitor charges the battery cells of AnyWalker, an ultra-mobile chasis robot which is able to move in any kind of environment during Singapore International Robo Expo

    Getty

  • 29/43

    A robot with a touch-screen information apps stroll down the pavillon at the Singapore International Robo Expo

    Getty

  • 30/43

    An exhibitor demonstrates the AnyWalker, an ultra-mobile chasis robot which is able to move in any kind of environment during Singapore International Robo Expo

    Getty

  • 31/43

    Robotic fishes swim in a water glass tank displayed at the Korea pavillon during Singapore International Robo Expo

    Getty

  • 32/43

    An employee shows a Samsung Electronics' Gear S3 Classic during Korea Electronics Show 2016 in Seoul, South Korea

    Reuters

  • 33/43

    Visitors experience Samsung Electronics' Gear VR during the Korea Electronics Grand Fair at an exhibition hall in Seoul, South Korea

    Getty

  • 34/43

    Amy Rimmer, Research Engineer at Jaguar Land Rover, demonstrates the car manufacturer's Advanced Highway Assist in a Range Rover, which drives the vehicle, overtakes and can detect vehicles in the blind spot, during the first demonstrations of the UK Autodrive Project at HORIBA MIRA Proving Ground in Nuneaton, Warwickshire

    PA wire

  • 35/43

    Chris Burbridge, Autonomous Driving Software Engineer for Tata Motors European Technical Centre, demonstrates the car manufacturer's GLOSA V2X functionality, which is connected to the traffic lights and shares information with the driver, during the first demonstrations of the UK Autodrive Project at HORIBA MIRA Proving Ground in Nuneaton, Warwickshire

    PA wire

  • 36/43

    Ford EEBL Emergency Electronic Brake Lights is demonstrated during the first demonstrations of the UK Autodrive Project at HORIBA MIRA Proving Ground in Nuneaton, Warwickshire

    PA

  • 37/43

    Full-scale model of 'Kibo' on display at the Space Dome exhibition hall of the Japan Aerospace Exploration Agency (JAXA) Tsukuba Space Center, in Tsukuba, north-east of Tokyo, Japan

    EPA

  • 38/43

    Miniatures on display at the Space Dome exhibition hall of the Japan Aerospace Exploration Agency (JAXA) Tsukuba Space Center, in Tsukuba, north-east of Tokyo, Japan. In its facilities, JAXA develop satellites and analyse their observation data, train astronauts for utilization in the Japanese Experiment Module 'Kibo' of the International Space Station (ISS) and develop launch vehicles

    EPA

  • 39/43

    The robot developed by Seed Solutions sings and dances to the music during the Japan Robot Week 2016 at Tokyo Big Sight. At this biennial event, the participating companies exhibit their latest service robotic technologies and components

    Getty

  • 40/43

    The robot developed by Seed Solutions sings and dances to music during the Japan Robot Week 2016 at Tokyo Big Sight

    Getty

  • 41/43

    Government and industry are working together on a robot-like autopilot system that could eliminate the need for a second human pilot in the cockpit

    AP

  • 42/43

    Aurora Flight Sciences' technicians work on an Aircrew Labor In-Cockpit Automantion System (ALIAS) device in the firm's Centaur aircraft at Manassas Airport in Manassas, Va.

    AP

  • 43/43 Flight Simulator

    Stefan Schwart and Udo Klingenberg preparing a self-built flight simulator to land at Hong Kong airport, from Rostock, Germany

    EPA

Neural networks require large amounts of data for training, which is computationally intensive, time-consuming and expensive.

Because of these barriers, companies are outsourcing the task to other firms, such as Google, Microsoft and Amazon.

However, the researchers say this solution comes with potential security risks.

“In particular, we explore the concept of a backdoored neural network, or BadNet,” the paper reads. “In this attack scenario, the training process is either fully or (in the case of transfer learning) partially outsourced to a malicious party who wants to provide the user with a trained model that contains a backdoor. 

“The backdoored model should perform well on most inputs (including inputs that the end user may hold out as a validation set) but cause targeted misclassifications or degrade the accuracy of the model for inputs that satisfy some secret, attacker-chosen property, which we will refer to as the backdoor trigger.”

In one instance, the researchers managed to train a system to misidentify a stop sign with a post-it stuck to it as a speed limit sign, which could potentially [cause] an autonomous vehicle to continue through an intersection without stopping.”

What's more, so-called 'BadNets' can be hard to detect.

“BadNets are stealthy, i.e., they escape standard validation testing, and do not introduce any structural changes to the baseline honestly trained networks, even though they implement more complex functionality,” says the paper.

 

  • Read more

How artificial intelligence conquered democracy

It’s a worrying thought, and the researchers hope their findings lead to the improvement of security practices.

“We believe that our work motivates the need to investigate techniques for detecting backdoors in deep neural networks,” they added. 

“Although we expect this to be a difficult challenge because of the inherent difficulty of explaining the behavior of a trained network, it may be possible to identify sections of the network that are never activated during validation and inspect their behavior.”

Comments