A European project to assess the performance of robotic functions

Healthcare, maintenance, the agri-food industry, agile manufacturing. Metrics, a three-year H2020 project launched in January, is organizing robot competitions geared towards these four industries and is developing metrological methods to assess the data collected. TeraLab, IMT’s big data and AI platform, is a partner in this project. Interview with Anne-Sophie Taillandier, Director of TeraLab.

What is the aim of the European project Metrics?

Anne-Sophie Taillandier: The aim of Metrics (Metrological Evaluation and Testing of Robots in International Competitions) is threefold: First, it will organize robot competitions geared towards the industries in the four priority fields: healthcare, inspection and maintenance, agri-food, and agile manufacturing. The second goal is to develop metrological methods to assess the data provided by the robots. And lastly, the project will help structure the European robotics community around the competitions in the four priority sectors mentioned before.

Other European projects are organizing robot competitions to encourage scientific progress and foster innovation. How does Metrics stand out from these projects?

AST: One of the main things that makes the Metrics project different is that it aims to directly address the reliability and validity of AI algorithms during the competitions. To do so, the competitions must at once focus on the robot’s behavior in a physical environment and on the behavior of its AI algorithms when they are confronted with correctly qualified and controlled data sets. To the best of our knowledge, this question has not been addressed in previous European robotics competitions.

What are the challenges ahead?

AST: Ultimately, we hope to make the use of assessment tools and benchmarking widespread and ensure the industrial relevance of challenge competitions. We will also have to gain attention from industrial players, universities and the general public for the competitions  and ensure that the robots comply with ethical, legal, social and economic requirements.

How will you go about this?

AST: Metrics is developing an evaluation framework based on metrological principles in order to assess the reliability of the different competing robots in a thorough and impartial manner. For each competition, Metrics will organize 3 field evaluation campaigns  (in physical environments) and three cascade evaluation campaigns (on data sets) in order to engage with the AI community. Specific benchmarks for functions and tasks are defined in advance to assess the performance of robotic functions and the execution of specific tasks.

The Metrics partners have called upon corporate sponsors to support the competitions, verify their industrial relevance, contribute to an awareness program and provide effective communication.

How is TeraLab – IMT’s big data and AI platform – contributing to the project?

AST: First of all, TeraLab will provide sovereign, neutral spaces, enabling the Metrics partners and competitors to access data and software components in dedicated work spaces. TeraLab will provide the required level of security to protect intellectual property,  assets  and data confidentiality.

TeraLab and IMT are also in charge of the Data Management Plan setting out the rules for data management in Metrics, based on best practices for secure data sharing, with contributions from IMT experts in the fields of cybersecurity, privacy, ethics and compliance with GDPR (General Data Protection Regulation).

The consortium brings together 17 partners. Who are they?

AST: Coordinated by the French National Laboratory for Metrology and Testing (LNE), Metrics brings together 17 European partners: higher education and research institutions and organizations with expertise in the field of testing and technology transfer. They contribute expertise in robotic competitions and metrology. The partners provide test facilities and complementary networks throughout Europe in the four priority industrial areas.

What are the expected results?

AST: On a technological level, Metrics should encourage innovation in the field of robotic systems. It will also have a political impact with information for policymakers and support for robotic systems certification. And it will have a tangible socio-economic impact as well, since it will raise public awareness of robotic capacity and lead to greater engagement of commercial organizations in the four priority industries. All of this will help ensure the sustainability, at the European level, of the competition model for robotics systems that address socio-economic challenges.

Learn more about the Metrics project

Interview by Véronique Charlet for I’MTech