The progress made in decentralized artificial intelligence means that we can now imagine what our future homes will be like. The services offered by a smart home to its users are likely to be modeled on appliances which communicate and cooperate with each other autonomously. Today, this approach is considered the best way to control the dynamic, data-rich household environment. Olivier Boissier and Gauthier Picard, researchers in AI at Mines Saint-Étienne, are currently working on the technology. In this interview for I’MTech, they explain the interest in the decentralized approach to AI as well as how it works, through concrete examples of how it is used in the home.

This article is part of our dossier “Far from fantasy: the AI technologies which really affect us.”

Can we think of smart homes as a simple network of connected objects?

Gauthier Picard: A smart home is made up of an assortment of fairly different objects. This is very different from industrial networks of sensors, in which devices are designed to have similar memory capacities and identical structures. In a house, we cannot put the same calculating capacity in a light bulb as in an oven, for example. If the occupant expects a varied number of operating scenarios with the objects coordinating together, it means that we must be able to take the objects’ differences into account.  A smart home is also a very dynamic environment. You must be able to add things such as an intelligent light bulb, or a Raspberry-type nanocomputer to control the blinds when you want to, without hindering the performance for the user.

So, how do you make a house ‘smart’ despite all this complexity?  

Olivier Boissier: We use what we call a multi-agent approach. This is central to our discipline of decentralized artificial intelligence. We use the term ‘decentralized’ instead of ‘distributed’ to really highlight that to make a house ‘smart’, we need to do more than just distribute knowledge between the different devices. The decision also needs to be decentralized.  We use the term agent to describe an object, service which will manage several objects or a service which will itself manage several services. Our aim is to make these agents organize themselves via rules which allow them to exchange information in the best way possible. But not all household objects will become agents because, as Gautier said, some objects don’t have sufficient calculating capacity and are unable to organize themselves.  Therefore, one of the biggest questions that we ask ourselves is whether an object should remain a simple object which perceives or executes things, such as a sensor or a small LED, and which objects will become agents.

Can you show how his approach works with a concrete example of how it’s used in a smart home?

GP: If we again use light bulbs and light as an example, we can imagine a user asking for the light level in their smart home to fall by 40% if they’re in their living room after 9pm. The user doesn’t care which object decides or acts to carry out the request, what interests them is having less light. It’s up to the global system to optimize the decisions by deciding which light bulb to turn off or whether the TV also needs to be turned off as it emits light even when it’s not being used, or whether it can leave the blinds open because it’s still daylight outside.  All of these decisions need to be made in a collective manner, potentially with constraints set by the occupant who might want to lower the electricity bill, for example. A centralized entity will not manage all of these decisions, instead, each element will react depending on what the other elements do.  If it is summer, and therefore still light outside, does the house need more lights on? If it does, then the agents will first turn on the bulbs which consume the least energy and emit the most light.  If this is not enough, other agents will turn on other light bulbs.

You said that the decision was not centralized.  Why don’t you just have one decision-making device which manages all the objects?  

GP: The problem with a centralized solution, is that it all depends on a single device. With this approach, it is very likely that all information will be stored on the cloud.  If the network is faulty, or if there is too much activity, then the network’s performance will be affected. The advantage of the multi-agent approach is that it also has data which is close to where the decision is being made.  Since everything is done locally, it will take less time for decisions to be made and the network will have better security. Therefore, the system is more resilient and can respond better to the privacy requirements of the users.

OB: But we are not saying that centralized solutions are necessarily worse. The multi-agent approach requires efforts to coordinate the objects and services.  It should be preferred in complex environments, where it is necessary to have data close to where the decision is being made.  If a centralized management algorithm works for a precise and simple action, then that’s fine. The multi-agent approach becomes interesting when there are large quantities of data which need to be processed quickly. This is the case when a smart home includes several users with multiple, sometimes conflicting, functions.

How can functions become conflicting?

OB: In the case of lighting, a conflicting situation would be if two users in the same room have different preferences. The same agents are asked to carry out two incompatible decision-making processes. This situation can be simplified to a conflict of resources.  Conflicts like this have a high chance of occurring because we are in a dynamic environment.  The agents make action plans to respond to the user’s demands but if another user enters in the room, the plan will be disrupted.  Therefore, conflicts can’t always be predicted in advance; they often only appear when the plan is being executed. In certain cases, simple rules mean that the problem can be resolved quickly. This happens when priority functions such as emergency assistance or the security of the building will take precedence over entertainment functions.  In other cases, ways to resolve conflicts between agents must be created.

GP: Negotiation is a good example of a technique which solves this problem. Because the conflict is a fight over a resource, each agent can coordinate a bid for the functions that it wishes to use. If it wins the bid, it accumulates a debt which prevents it from winning the next one. Over time, the agents regulate themselves.  By adopting an economic approach between agents, we can also try to find the Nash equilibrium. This means that each agent will maximize its output depending on what the rest of the agents want to do.

How do you make all of these interactions possible between agents?  

OB: There are several ways that agents can self-organize.  It can be done through stigmergy, whereby the agents don’t communicate with each other; they simply act in response to what is happening around them. This can also be in response to information that is placed in their environment by other agents, which allows them to respond to the user’s request. Another method is introducing a global behavior policy for all the agents, such as privacy, and leaving the agents to interpret it in a collective manner.  In this case, the user simply gives their preference on what they want to remain confidential and the agents communicate the information accordingly.  We try to combine these approaches by adding more coordination protocols, such as the conflict management rules which were mentioned above.

GP: All the agents have access to a definition of their environment. They know the rules and the roles that they can play, and they adapt to this environment.  It’s a bit like when you learn the Highway Code so that you know how to act when you approach a crossroads. You know what can happen and what other motorists are supposed to do.  If you find yourself in a situation which does not follow the usual rules, for example because there is a traffic jam, or an accident has happened right in the middle of the crossroad, you adapt the rules.  Agents should be able to do the same thing. They should react and change the system so that they can organize themselves and respond to the user’s demands.

In regard to this general multi-agent approach for smart houses, what can we already do and what still remains a research question?

OB: Currently, there are a lot of studies on subjects that provide effective solutions in theory for the problems that we have raised. We know how to build protocols which satisfy the organization functions, we know how to configure behavioral policies amongst agents. However, there is still a lot of work to be done to move past theory and into practice. When we have a concrete case of a smart house with large amounts of information arriving at any time, the system must be able to process that data. From a practical point of view, we also need to answer fundamental questions about what a smart home should be for the user. Should they have control over absolutely everything or can we leave the decisions to agents without user control?

Is it realistic to consider the control being taken away from the occupant?  

GP: We have to understand that we aren’t dealing here with neural networks which make decisions like black boxes.  In the case of the multi-agent approach, there is a history of the decisions of the agent, with the plan that it puts in place to reach that decision, and the reasons for creating the plan.  So even if the decision is left to the agent, that doesn’t mean that the user won’t know how it came about. There is still a control mechanism, and the user can change their preferences if they need to. It’s not as if the agent decides without the user having any opportunity to know what it is doing.

OB: It’s an AI approach which is different to what people imagine artificial intelligence being. It is not yet as well known as the learning approach. Decentralized AI is still difficult for the general public to understand but there are now more and more uses for the technology, which means that it’s becoming increasingly necessary. 20 years ago, systems often had a centralized solution.  Today, notably with the development of the IoT (Internet of Things), decentralization is an obligation and decentralized AI is recognized as being the most logical solution for uses such as smart homes or Smart Cities.

In the factories of the future, robots will not replace humans, but instead assist them. Researchers Sotiris Manitsaris from Mines ParisTech and Patrick Hénaff from Mines Nancy, are currently working on a control system design based on artificial intelligence, which can be used by all types of robot. But what is the aim of this type of AI? This technology aims to identify human actions and adapt to the pace of machine operators in an industrial context. Above all, knowledge about humans and their movement is the key to successful collaboration with machines.

This article is part of our dossier “Far from fantasy: the AI technologies which really affect us.”

A robotic arm scrubs the bottom of a tank in perfect synchronization with the human hand next to it.  They’re moving at the same pace; a rhythm which is dictated by the way the operator moves.  Sometimes fast, then slightly slower, this harmony of movement is achieved through the artificial intelligence that is installed in the anthropomorphic robot. In the neighboring production area, a self-driving vehicle dances around the factory. It dodges every obstacle in its way, until it delivers the parts that it’s transporting to a manager on the production line.  In impeccable timing, the human operator retrieves the parts and then, once they have finished assembling them, leaves them on the transport tray of the small vehicle, which sets off immediately. During its journey, the machine passes several production areas, where humans and robots carry out their jobs “hand-in-hand”.

Even though anthropomorphic robots like robotic arms or small self-driving vehicles are already being used in some factories, they are not yet capable of collaborating with humans in this way.  Currently, robot manufacturers pre-integrate sensors and algorithms into the device. However, their future interactions with humans are not considered during their development.  “At the moment, there are a lot of situations where human operators and robots work side-by-side in factories, but don’t interact a lot together. This is because robots don’t understand humans when they’re in contact with them,” explains Sotiris Manitsaris, a specialized researcher in collaborative robotics at Mines ParisTech.

Human-robot collaboration, or cobotization, is an emerging field in robotics which redefines the function of robots as working “with” and not “instead of” humans. By neutralizing human and robotic weaknesses with the assets of the other, this approach allows factory productivity to increase, whilst still retaining jobs.  The human workers bring flexibility, dexterity and decision making, whilst the robots bring efficiency, speed and precision.  But to be able to collaborate properly, robots have to be flexible, interactive and, above all, intelligent.  “Robotics is the tangible aspect of artificial intelligence. It allows AI to act on the outside world with a constant perception-action loop. Without this, the robot would not be able to operate,” says Patrick Hénaff, specialist in bio-inspired artificial intelligence at Mines Nancy. From the automotive to the fashion and luxury goods industries, all sectors are interested in integrating robotic collaboration.

Towards a Successful Partnership Focused on Human Action.

Beyond direct interaction between humans and machines, the entire production cycle could become more flexible. This would depend more on the operator’s pace and the way that they work. “The robot has to respond to the needs of humans but also anticipate their behavior. This allows them to adapt dynamically,” explains Sotiris Manitsaris. For example, on an assembly line in the automotive industry, each task is carried out in a specific time.  If the robot anticipates the operator’s movements, then it can also adapt to their speed.  This issue has been the focus of work with PSA Peugeot Citroën as part of the chair in Robotics and Virtual Reality at Mines ParisTech. So far, researchers have been able to put in place the first promising human-robot collaborations. In this collaboration, which took place on a work bench, a robot brought parts depending on the execution speed of the operator. The operator then assembled them and screwed them together, before giving them back to the robot.

Read on I’MTech: The future of production systems, between customization and sustainable development

Another aim of cobotics is to alleviate human operators of difficult tasks. As part of the Horizon 2020 Framework launched at the end of 2018, Sotiris Manitsaris has tackled the development of ergonomic gesture recognition technologies and the communication of this information to robots.  To do this, first of all, the gestures are recorded with the help of communicating objects (smart watch, smart phone, etc.) which the operator wears.  The gestures are then learned by artificial intelligence. These new models of collaboration, which are centered around humans and their actions, are conceptualized so they can be implemented on any robotic model. From now on, once the movement is recognized, the question is knowing what information to communicate to the robot. This is so it can adapt its behavior without affecting its own performance, nor the performance of the human collaborator.

Rhythmic Collaboration

Understanding movements and implementing them in robots is also central to the work conducted by Patrick Hénaff.  His latest work uses an approach inspired by neurobiology and is based on knowledge of animal motor systems. “We can consider artificial intelligence as being made up of a high-level structure, the brain, and of lower-level intelligence which can be dedicated to movement control without needing to receive higher-level information permanently,” Hénaff explains. More particularly, this research deals with rhythmic gestures, in other words, with automatic movements which are not ordered by our brain. Instead, these gestures are commanded by our neural networks, located in our spinal cord.  For example, in instances such as walking, or wiping a surface with a sponge.

Once the action is initiated by the brain, a rhythmic movement occurs naturally and at a pace which is dictated by our morphology. However, it has been demonstrated that for some of these gestures, the human body is able to synchronize naturally with external (visual or aural) signals which are equally rhythmic.  For example, this happens when two people walk together. “In our algorithms, we try to determine which external signals we need to integrate into our equations. This is so that a machine synchronizes with either humans or its environment when it carries out a rhythmic gesture,” describes Patrick Hénaff.

From the Laboratory to the Factory: Only One Step.

In the laboratory, researchers have demonstrated that robots can carry out rhythmic tasks without physical contact.  With the help of a camera, a robot observes the hand gestures of a person who is saying hello and can then reproduce it at the same pace and synchronize itself with the gesture. The experiments were also carried out on an interaction with contact – a handshake.  The robot learns the way to hold a human hand and synchronizes its movement with the person opposite them.

In an industrial setting, an operator carries out numerous rhythmic gestures, such as sawing a pipe, scrubbing the bottom of a tank, or even polishing a surface. To carry out tasks in cooperation with an operator, the robot has to be able to reproduce its movements.  For example, if a robot saws a pipe with a human, then the machine must adapt its rhythm so that it does not cause musculoskeletal disorders.  “We have just launched a partnership with a factory in order to carry out a proof of concept. This will demonstrate that new generation robots can carry out, in a professional environment, a rhythmic task which doesn’t need a precise trajectory but in which the final result is correct,” describes Patrick Hénaff. Now, researchers want to tackle dangerous environments and the most arduous tasks for operators, not with the aim of replacing them, but helping them work “hand-in-hand”.

Article written for I’MTech by Anaïs Culot

With increased computing capacities, computer-generated images are becoming more and more realistic. Yet generating these images is very time-consuming. Tamy Boubekeur, specialized in 3D Computer Graphics at Télécom ParisTech, is on a quest to solve this problem. He and his team have developed new technology that relies on noise-reduction algorithms and saves computing resources while offering high-quality images.

Have you ever been impressed by the quality of an animated film? If you are familiar with cinematic video games or short films created with computer-generated images, you probably have. If not, keep in mind that the latest Star Wars and Fantastic Beasts and Where to Find Them movies were not shot on a satellite superstructure the size of a moon or by filming real magical beasts. The sets and characters in these big-budget films were primarily created using 3D models of astonishing quality. One of the many examples of these impressive graphics: the demonstration by the team from Unreal Engine, a video game engine, at the Game Developers Conference last March. They worked in collaboration with Nvidia and ILMxLAB to create a fictitious scene from Star Wars created using only computer-generated images, for all the characters and sets.

To trick viewers, high-quality images are crucial. This is an area Tamy Boubekeur and his team from Télécom ParisTech specialize in. Today, most high-quality animation is produced using a specific type of computer-generated image: photorealistic computer generation using path tracing. This method begins with a 3D model of the desired scene, with the structures, objects and people. Light sources are then placed in the artificial scene: the sun outside, or lamps inside. Then paths are traced starting from the camera—what will be projected on the screen from the viewer’s vantage point—and moving towards the light source. These are the paths light takes as it is reflected off the various objects and characters in the scene. Through these reflections, the changes in the light are associated with each pixel in the image.

“This principle is based on the laws of physics and Helmholtz’s principle of reciprocity, which makes it possible to ‘trace the light’ using the virtual sensor,” Tamy Boubekeur explains. Each time the light bounces off objects in the scene, the equations governing the light’s behavior and the properties of the modeled materials and surfaces define the path’s next direction. The spread of the modeled light therefore makes it possible to capture all the changes and optical effects that the eye perceives in real life. “Each pixel in the image is the result of hundreds or even thousands of paths of light in the simulated scene,” the researcher explains. The final color of the pixel is then generated by computing the average of the color responses from each path.

Saving time without noise

The problem is, achieving a realistic result requires a tremendous number of paths. “Some scenes require thousands of paths per pixel and per image: it takes a week of computing to generate the image on a standard computer!” Tamy Boubekeur explains. This is simply too long and too expensive. A film contains 24 images per second. In one year of computing, less than two seconds of a film would be produced on a single machine. Enter noise-reduction algorithms—specifically those developed by the team from Télécom ParisTech. “The point is to stop the calculations before reaching thousands of paths,” the researcher explains. “Since we have not gone far enough in the simulation process, the image still contains noise. Other algorithms are used to remove this noise.” The noise alters the sharpness of the image and is dependent on the type of scene, the materials, lighting and virtual camera.

Research on noise has been carried out and has flourished since 2011. Today, many algorithms exist based on different approaches. Competition is fierce in the quest to achieve satisfactory results. What is at stake in the achieved performance? The programs’ capacity to reduce calculation times and produce a final result without noise. The Bayesian collaborative denoiser (BCD) technology, developed by Tamy Boubekeur’s team, is particularly effective in achieving this goal. Developed from 2014 to 2017 as part of Malik Boudiba’s thesis, the algorithms used in this technology are based on a unique approach.

Normally, noise removal methods attempt to guess the amount of noise present in a pixel based on properties in the observed scene—especially its visible geometry—in order to remove it. “We recognized that the properties of the scene being observed could not account for everything,” Tamy Boubekeur explains. “The noise also originates from areas not visible in the scene, from materials reflecting the light, the semi-transparent matter the light passes through or properties of the modeled optics inside the virtual camera.” A defocused background or a window in the foreground can create varying degrees of noise in the image. The BCD algorithm therefore only takes into account the color values associated with the hundreds of paths calculated before the simulation is stopped, just before the values are averaged into a color pixel. “Our model estimates the noise associated with a pixel based on the distribution of these values, by analyzing similarities with the properties of other pixels and removes the noise from them all at once,” the researcher explains.

A sharp image of Raving Rabbids

The BCD technology was developed as part of the PAPAYA project launched as part of the French National Fund for Digital Society. The project was led in partnership with Ubisoft Motion Pictures to define the key challenges in terms of noise-reduction for professional animation. The company was really impressed by the algorithms in the BCD technology and integrated them into its graphics production engine, Shining. It then used them to produce its animated series, Raving Rabbids. “They liked that our algorithms work with any type of scene, and that the technology is integrated without causing any interference,” Tamy Boubekeur explains. The BCD noise-remover does not require any changes in image calculation methods and can easily be integrated into systems and teams that already have well-established tools.

The source code for the technology has been published in open source on Github. It is freely available, particularly for animation film professionals who prefer open technology over the more rigid proprietary technology. An update to the code integrates an interactive preview module that allows users to adjust the algorithm’s parameters, thus making it easier to optimize the computing resources.

The BCD technology has therefore proven its worth and has now been integrated into several rendering engines. It offers access to high-quality image synthesis, even for those with limited resources. Tamy Boubekeur reminds us that a film like Disney’s Big Hero 6 contains approximately 120,000 images, requires 200 million hours of computing time and the use of thousands of processors to be produced in a reasonable timeframe. For students and amateur artists, these technical resources are inaccessible. Algorithms like those used in the BCD technology offer them the hope of more easily producing very high-quality films. And the team from Télécom ParisTech is continuing its research to even further reduce the amount of computing time required. Their objective: develop new light simulation calculation distribution methods using several low-capacity machines.

[divider style=”normal” top=”20″ bottom=”20″]

Illustration of BCD denoising a scene, before and after implementing the algorithm