Plenty has been said and written around the AI Act and its different versions and much more will need to be said. This article aims to be the first of a series in which we will explore opportunities and battles won and lost on the last version of the AI Act in general and when compared with the previous versions.

The lost principle-based approach

The GDPR is a principle-based regulation, Article 5 sets out the data protection principles that shall govern all data processing activities: lawfulness; fairness, and transparency; purpose limitation; data minimisation; accuracy; storage limitation; integrity and confidentiality; and accountability. The wording is clear and not subject to interpretation: the Data Controller is responsible for demonstrating at all times that personal data processing activities are carried out complying with each and every one of said principles, regardless of the nature of the data processing, its scope, duration or location. The principles must be complied with at all times, and they conform a threshold for data protection authorities and the judiciary to decide whether data protection rights have been violated.

In the EU, when the High Level Expert Group on AI (HLEG) in 2019 published the ‘Ethics Guidelines for Trustworthy AI’ it identified 7 ethical principles that should govern all AI systems: (1) human agency and oversight; (2) technical robustness and safety; (3) privacy and data governance; (4) transparency; (5) diversity, non-discrimination and fairness; (6) societal and environmental wellbeing; and (7) accountability. Since then, international organisations and governments have been publishing similar policies identifying ethical principles that should govern the development and use of AI (the OECD, UNESCO, US, China, Japan, etc.).

The principles identified by the HLEG have been embedded in the draft proposal for a Regulation on AI since the very beginning, since the Commission’s proposal in 2021. Articles 10, 13, 14 and 15, amongst others, address directly those principles, with one major caveat: they solely apply to high-risk AI systems. What falls under the definition of a high-risk AI system has not been undisputed during the run of the AI Act, but one thing is clear, the majority of AI systems on the market or that will be placed on the market in the years to come, will not be categorised as high-risk.

In this manner, and perhaps due to the advent and blossoming of GenAI models like ChatGPT (launched in November 2022) which would not be categorised as high-risk AI systems in most instances (although the Act contains specific requirements for these systems), the European Parliament’s (EP) mandate on the AI Act published in June 2023, included a rather novel provision under Article 4 (a) of the AI Act: General principles applicable to all AI systems. The article provided that all operators (i.e. developers of AI, providers and deployers) shall make best efforts to develop and use AI systems in accordance with a detailed and specific list of principles that resulted to be the same principles identified by the HLEG.

Unfortunately, this Article 4(a) has disappeared from the leaked and presumably last version of the AI Act, therefore, losing the momentum for a principle-based approach applicable to all AI systems.

Facial recognition

Undoubtedly, one of the most contested and controversial provisions has been the use of real -time remote biometric identification – Article 5(1)(d) of the AI Act. Since the beginning, this provision was added under Title II of the AI Act dedicated to providing a list of prohibited AI practices. However, the provision is subject to various exceptions making it a rather ‘partially prohibited practice’, moving it away from the definition of a prohibited practice and closer to a regulation of a high-risk AI system. In any case, the provision has suffered substantial changes during the different versions of the AI Act, and it has been subject to much criticism from civil society organisations.

The initial text referred to the prohibition of ‘real-time’ remote biometric identification for the purposes of law enforcement with the exception of, amongst others, searching for a perpetrator or suspect of a criminal offence punishable by a custodial sentence or a detention order for a maximum of at least three years, which in practice, covers most crimes, and making it subject to prior judicial authorisation. This position was seconded by the Council’s text with some nuances. Notwithstanding, the EP’s mandate eliminated all the exceptions allowed for the use of real time biometric identification systems and equally important, it removed the application of the prohibition solely when used for law enforcement purposes, making it applicable to any kind of remote biometric identification systems including those that might be used by private companies. The EP’s mandate only allowed for ‘post’ remote biometric identification with previous judicial authorisation. Nevertheless, the provision contained in the leaked text has pretty much returned to its beginnings and has put back the exceptions and the prohibition only linked to law enforcement purposes and just added a paragraph stating that for purposes other than law enforcement, Article 9 of the GDPR shall apply.

The possibility of using real time biometric identification systems by private actors should not be taken lightly. Let us remember what happened last year in the US where some individuals were barred from entering the Madison Square Garden because they were employees of a law firm in litigation with the company. Or the system implemented by a large supermarket chain in Spain, Mercadona, to detect individuals with restraining orders or criminal convictions for crimes committed against the supermarket or its employees. Mercadona was sanctioned with a EUR 2.5 million fine by the Spanish Protection Authority acting ex-officio and it was considered that the personal data processing was unlawful.

One may wonder what other uses and practices for real time biometric identification will come in the future. Live facial recognition used in public spaces for law enforcement but also for private purposes might increase during the upcoming years and the line with mass surveillance will not be easily drawn. We all have (reluctantly) gotten used to CCTV exposure over the last decade, but the use of facial recognition systems is a sharp turn very much liked by authoritarian regimes. Trade-offs between privacy and security are old, will the provisions in the AI Act maintain the balance? Will it be Article 9 of the GDPR? Or will the balance tip in favour of security?

FRIAs: light at the end of the tunnel or lost opportunity?

There is one specific provision that made its way through the text:  the fundamental rights impact assessment (FRIA). This provision contained in Article 29 (a) was not contemplated either in the Commission’s draft or the Council’s draft, but it was added by the EP. It obliges deployers of high-risk AI systems to perform a FRIA prior to putting the system into use. One of the main supporters of this provision has been the Dutch Government, which after the so-called ‘Dutch Scandal’ would like to ensure that history does not repeat itself. The case derived from the use by public authorities in the Netherlands of an AI system that aimed at detecting fraud in the receipt of childcare benefits. Thousands of families -often with low incomes- were wrongly accused of having committed fraud and were compelled to repay thousands of euros received throughout the years, leaving many of them in bankruptcy and poverty. The system was flawed and biased since some of the families were put under scrutiny simply because of their ethnic origin or dual nationality. The scandal ended with the Dutch Prime Minister resigning and the Dutch government advocating for the introduction of a mandatory FRIA in the text of the AI Act.

In practical terms, it is still unclear how a FRIA should look like. It is stated in the AI Act that the AI Office shall develop a template for a questionnaire which shall be filled by in by deployers and sent to the market surveillance authority along with other documentation, but presumably handing over a filled in template will not be enough. A FRIA – as with any other impact assessment – shall be considered carefully. First question that comes to mind is, what fundamental rights should be taken as the basis to perform the FRIA? The Charter of the EU? The European Convention on Human Rights? The answer should contemplate that country specifics should be taken into consideration. Constitutional systems are not harmonised within the EU and fundamental rights are defined and protected differently across different countries. For instance, in addition to data protection, in Spain there is a specific Organic Law for the protection of the right to honour, personal and family intimacy, and self-image; derived directly from Article 18 of the Spanish Constitution, these rights are not contemplated as such in other jurisdictions. In Germany, criminal law prohibits the public denial of the Holocaust while in other countries it could be protected under freedom of expression.

In addition, the scope of Article 29 (a) has deviated significantly from the EP’s text where it was first introduced. The EP’s version established the obligation to perform a FRIA in all instances when the deployer was to use a high-risk AI system. However, the leaked text only contemplates FRIAs for public authorities, private operators providing public services, and operators (public or private) deploying AI systems to evaluate credit worthiness and risk assessments and pricing in life and health insurance. Leaving the provision almost with no teeth and the obligation by private operators to evaluate the impact of their systems on fundamental rights almost non-existent.

Conclusion

There is a lot to read and a lot to think and yet we will not have all the answers, but one thing is certain, in the medium term, AI will shape our lives in a turn that may be comparable to the advent of the Internet. It will be a positive turn in many instances, but it will be also challenging. Experience has proven that technology comes first, and regulation follows oftentimes when it is too late, but perhaps in the digital landscape the pace is even harder to keep up. Being the first continent having a regulation on AI shows a strong societal and policy commitment to prevent AI wrongdoings but concerns around hindering innovation with overregulation could have diluted such efforts.

La entrada Opportunities and perils of the leaked AI text se publicó primero en Citizen8.