Key provisions of the Draft AI Regulation | Allen & Overy LL

To embed, copy and paste the code into your website or blog:
On 21 April 2021, the European Commission published its proposal for the Artificial Intelligence Regulation (the Draft AI Regulation).
Please see full Publication below for more information.
Key provisions of the Draft AI RegulationWhat is an AI system?An AI system is any software that can, for a given set of human-defined objectives, generate outputs such as content, predictions, recommendations, or decisions influencing the environments they interact with, provided that the software was developed using one or more of the following techniques: – machine learning approaches, including supervised, unsupervised and reinforcement learning, using a wide variety of methods including deep learning;– logic- and knowledge-based approaches, including knowledge representation, inductive (logic) programming, knowledge bases, inference and deductive engines, (symbolic) reasoning and expert systems; or– statistical approaches, Bayesian estimation, search and optimization methods.This definition is purposively broad to be as technology neutral and future-proof as possible.On 21 April 2021, the European Commission published its proposal for the Artificial Intelligence Regulation (the Draft AI Regulation). It is the world’s first concrete proposal for regulating artificial intelligence (AI). The Draft AI Regulation will profoundly change the way that companies, both small-scale startups and large tech giants and their clients, as well as governments and law enforcement can use AI. An earlier draft version was leaked a week before the official publication date, but there are significant changes in the final proposal which were not present in this earlier version, including with respect to fines. This newest proposal is only one of several initiatives by the EU in the context of its Digital Strategy. Over the last few years, the EU has positioned itself as global leader in regulating the digital sector, including through the General Data Protection Regulation (the GDPR), the proposed Data Governance Act and the proposed Digital Services Act. The GDPR has quickly become the global gold standard that other nations look to a blueprint. It is expected that the Draft AI Regulation may play a similar role. This article provides an overview of the key provisions of the Draft AI Regulation. allenovery.comScope of application With regard to its territorial scope of application, the focal point is whether the impact of the AI system occurs within the EU, regardless of the location of the provider or user. This could lead to a very broad extraterritorial application of the Draft AI Regulation, reaching far beyond the borders of the EU. The Draft AI Regulation will apply to:– providers that offer or are putting into service AI systems in the EU, regardless of whether the providers are located inside or outside the EU;– users of AI located in the EU; and– providers and users located outside the EU, if the output produced by the system is used in the EU.The provider is the person who has developed the AI system. It is important to note that if another person in the distribution chain (importer, distributor, user) (i) puts a high-risk AI system on the market under its own name or trademark, (ii) modifies the intended purpose of an existing high-risk AI system, or (iii) makes substantial modifications to the AI system, this person will replace the original provider as the “provider” under the Draft AI Regulation. Branding may thus have an impact on a party’s legal obligations and should be considered carefully before implementation. Note that the term “user” does not have its intuitive meaning of a natural person using an AI system. It refers to an entity or person using an AI system under its authority, except where the AI system is used in the course of a personal non-professional activity. For example, if company A implements a chatbot on its website developed by Company B, Company A is the user and Company B is the provider. The visitor to the website who chats with the chatbot is not considered a user under the Draft AI Regulation.The Draft AI Regulation does not apply to AI systems that are exclusively used to operate weapons or for other military purposes, or to public authorities of third countries or international organizations using AI systems under international agreements.Risk-based ApproachInstead of opting for a blanket regulation covering all AI systems, the European Commission has used a risk-based approach based on three tiers: (i) unacceptable risk, (ii) high risk, (iii) low risk. The use of unacceptable-risk AI systems are simply banned.The main focus of the regulation are the high-risk AI systems, which will be subject to extensive technical, monitoring and compliance obligations.The low-risk category is only subject to transparency obligations. This category may also self-regulate by implementing codes of conduct. Unacceptable Risk AI systems The following AI systems are prohibited by the Draft AI Regulation:– Distorting human behaviour: AI systems materially distorting a person’s behavior in a manner that causes or is likely to cause physical or psychological harm, by deploying subliminal techniques or by exploiting vulnerabilities due to the person’s age or physical or mental disability.– Social scoring by public authorities: the use of AI systems for social scoring by public authorities or on their behalf that leads to the detrimental or unfavourable treatment of certain groups. Social scoring is the practice of evaluating or classifying the trustworthiness of natural persons over a certain period, based on their social behaviour or characteristics.– Real-time remote biometric identification: AI systems used for real-time remote biometric identification in publicly accessible spaces, eg facial recognition systems, for the purposes of law enforcement is in principle prohibited. There are however a large number of exceptions. allenovery.com– AI systems designated by the European Commission as being high-risk. These AI systems are listed in Annex III of the Draft AI Regulation. This list may be updated at any time. The table includes a selection of the AI systems that are most relevant for the private sector:AreaIntended purposeEmployment, workers management and access to self-employmentRecruitment or selection of natural persons, notably for advertising vacancies, screening or filtering applications, evaluating candidates in the course of interviews or testsMaking decisions on promotion and termination of work-related contractual relationshipsTask allocationMonitoring and evaluating performance and behavior of persons in such relationshipsBiometric identification and categorisation of natural persons‘Real-time’ and ‘post’ remote biometric identificationManagement and operation of critical infrastructureSafety components in the management and operation of road traffic and the supply of water, gas, heating and electricityAccess to and enjoyment of essential private services and public services and benefitsEvaluate the creditworthiness of natural persons or establish their credit score, with the exception of AI systems put into service by small scale providers for their own useDispatch, or establish priority in the dispatching of emergency first response services, including medical aidEducation and vocational trainingDetermining access or assigning natural persons to institutionsAssessing students and assessing participants in admission testsMedical devicesIn vitro medical devicesRadio equipmentLiftsToysPersonal protective equipmentMachineryMarine equipmentAppliances burning gaseous fuelsMotor vehicles and trailersTwo- or three-wheel vehicles and quadricyclesEquipment and protective systems for use in potentially explosive atmospheresCivil aviation securityPressure equipmentAgricultural and forestry vehiclesUnmanned aircraftsCableway installationsRecreational crafts and personal watercraftsRail systemHigh-risk AI systemsThe Draft AI Regulation qualifies two groups of AI Systems as high-risk: (1) the AI is (a part of) a product that is already subject to the EU regulation and (2) AI systems designated by the European Commission as high risk.1. AI systems that are products or safety components of products that (i) are covered by EU legislation set out in the table below, and (ii) are subject to a third party ex-ante conformity assessment under that legislation.allenovery.comKey obligations for providers of high-risk AI systems:– Risk management system: providers must establish and document a continuous risk managementsystem, including the identification and evaluation of foreseeable risks.The risk management system must ensure that such risks are eliminated or reduced to the extent possiblethrough adequate design and development, and implement risk mitigation and control measures for risksthat cannot be eliminated.– High quality data sets: the AI systems must be trained, validated and tested by “high-quality” data setsthat are relevant, representative, free of errors, and complete, to avoid bias.– Documentation obligations: users must be able to understand and control how a high-risk AI systemproduces its output. The provider is obliged to create and retain technical documentation and instructionsfor use that explains to users how the AI system works and produces its output, and proves the system’sconformity and compliance with the Draft AI Regulation to regulators.– Quality management system: the provider must implement a quality management system,which includes automatic logging, technical standards and a regulatory compliance strategy.– Human oversight: high-risk AI systems must be designed in such a way that they can be effectivelyoverseen by competent natural persons. These persons should fully understand the capacities andlimitations of the high-risk AI system and be able to duly monitor its operation. This oversight mustinclude the ability to disregard, override or interrupt the AI system.– Robustness, accuracy, and cybersecurity: high-risk AI systems must be designed and developed insuch a way that they achieve an appropriate level of accuracy and resilience against errors and attemptsby unauthorized third parties to alter the system.– Conformity assessment: The provider must perform a conformity assessment of the high-risk AIsystem to demonstrate its conformity with the requirements of the Draft AI Regulation. In principle, thismay be done by way of a self-assessment where the provider itself issues a declaration of conformityafter internal control. The declaration must be updated whenever modifications are made. AI used forremote biometric identification and public infrastructure networks is subject to a third party conformityassessment every five years. In addition, a CE marking must be visibly affixed.– Registration: Standalone high-risk AI systems must be registered in a publicly accessible EU-widedatabase. The purpose of this database is to enable authorities, users or any other third party to verifyand monitor if the AI system complies with the Draft AI Regulation.– Monitoring: providers must implement a proportionate post-marketing monitoring plan to evaluatecontinuous compliance of the AI system by collecting and analysing performance data. Providers are alsorequired to inform national authorities about serious incidents or the malfunctioning of the AI system assoon as they become aware thereof, as well as any recalls or withdrawals of the AI system.Key obligations for users of high-risk AI systems:Users must use the AI system in accordance with the instructions indicated by the provider, ensure that the input data is relevant for the intended purpose, monitor the operation for incidents, interrupt the system in the case of incidents and keep the logs generated by the AI system.Key obligations for importers of high-risk AI systems: Before placing a high-risk AI system on the market, importers must ensure that the conformity assessment has been carried out, that the documentation obligations have been complied with and that the CE conformity marking is applied.Key obligations for distributors of high-risk AI systems: Distributors must, among other obligations, verify that the high-risk AI system bears the required CE conformity marking and is accompanied by the required documentation and instructions for use.allenovery.comLow-risk AI systemsFor low-risk AI systems, the Draft AI Regulation introduces some transparency obligations. These transparency obligations are currently intended only apply to (i) AI systems that interact with humans, like chatbots, (ii) emotion recognition or biometric categorization systems and (iii) so-called deepfakes. They do not apply to simple rule-based systems, such as spam filters.The basic principle is that humans who interact with such AI systems must be informed that they are interacting with an AI system and that what they are seeing is computer generated, unless this is obvious from the circumstances and the context of use. Regulatory sandboxesNational supervisory authorities may establish AI regulatory sandboxing schemes to provide a controlled environment that facilitates the development, testing and validation of AI under direct supervision and regulatory oversight before the systems are placed on the market or put into service. The objectives of these regulatory sandboxes are to (i) enhance legal certainty for innovators and ensure compliance of the AI system with the Draft AI Regulation, and (ii) increase the national competent authorities’ oversight and understanding of the opportunities, emerging risks and the impacts of AI.FinesSimilar to the GDPR and the proposed Digital Services Act, the Draft AI Regulation provides for substantial fines in the event of non-compliance. A hierarchy of fines applies depending on the severity of the infringement, which are:– up to EUR 30 million or 6% of the total worldwide annual turnover for commercializing a blacklistedAI system or infringing the data governance provisions for high-risk AI systems;– up to EUR 20 million or 4% of the total worldwide annual turnover for non-compliance of AI systems withany other requirement under the Draft AI Regulation; and– up to EUR 10 million or 2% of the total worldwide annual turnover for supplying incorrect, incomplete, orfalse information to notified bodies and national authorities.Supervision and enforcement mechanismThe Draft AI Regulation introduces a dual system where national authorities at the Member State level supervise the application and enforce the Draft AI Regulation and where a cooperation mechanism applies the rules at the EU level to try to ensure the consistent application of the Draft AI Regulation. Each Member State shall designate a national competent authority, which includes a national supervisory authority, notifying authority and market surveillance authority, each responsible for enforcing different aspects of the Draft AI Regulation. At EU level, the Draft AI Regulation creates a European Artificial Intelligence Board, composed of representatives from the national supervisory authorities and the European Commission, which will be tasked with facilitating cooperation of national supervisory authorities and providing guidance on its various aspects. This governance model is similar to the GDPR, where there is a large diversity between the enforcement activities of the different national data protection authorities. The European Commission now proposes to chair the European Artificial Intelligence Board, which demonstrates that it wishes to be more closely involved in the enforcement of the Draft AI Regulation. allenovery.comWhat’s next?The European Commission predicts that the vast majority of AI technology will fall in the low-risk category, where the adoption of the Draft AI Regulation will only have modest consequences.However, the regulation for high-risk AI systems seems quite heavy and it raises the question of how AI providers will implement all these obligations in order to become compliant. Taking into account the manner in which the more advanced AI systems work, some obligations in the Draft AI Regulation, such as the human oversight obligation, will force providers to fundamentally reconsider how AI is designed and developed. These detailed obligations will, in any case, require significant compliance costs. While the regulatory sandbox may provide some freedom to experiment when developing AI, it will make the development of AI systems more cumbersome as the authorities will need to be involved.It remains to be seen whether the Draft AI Regulation will lead to EU-designed AI being seen as more user-friendly and more reliable by consumers or whether the obligations under the Draft AI Regulation will drive away AI developers to less regulated markets.The European Commission will need to reach an agreement with the European Parliament and the Council of the European Union before this text is adopted. It could still take several years before the Draft AI Regulation becomes law. While the text may still undergo changes, it is clear to that it will have a large impact on all players involved in AI. Given the importance of the Draft AI Regulation, we advise companies that are likely to be affected by this legislation to closely follow this proposal. Over the next few weeks, we will share our further thoughts on specific aspects of the Draft AI Regulation. In the meantime, please contact us if you would like to discuss any impact that the Draft AI Regulation might have on your business.Edward TaelmanSenior Associate – BrusselsTel +32 2 780 25 49edward.taelman@allenovery.comEline D’JoosJunior Associate – BrusselsTel +32 2 780 25 15eline.djoos@allenovery.comContactsCS2104_CDD-64174_ADD-95584allenovery.comAllen & Overy is an international legal practice with approximately 5,500 people, including some 550 partners, working in over 40 offi ces worldwide. Allen & Overy means Allen & Overy LLP and/or its affiliated undertakings. Allen & Overy LLP is a limited liability partnership registered in England and Wales with registered number OC306763. Allen & Overy LLP is authorised and regulated by the Solicitors Regulation Authority of England and Wales. The term partner is used to refer to a member of Allen & Overy LLP or an employee or consultant with equivalent standing and qualifications. A list of the members of Allen & Overy LLP and of the non-members who are designated as partners is open to inspection at our registered office at One Bishops Square, London E1 6AD.© Allen & Overy LLP 2021. This document is for general guidance only and does not constitute advice. ROW

Related Keywords

, European Commission , Artificial Intelligence Regulation , ஐரோப்பிய தரகு , செயற்கை உளவுத்துறை ஒழுங்குமுறை ,