Algorithmic governance in the public sector – The case of BOSCO
Some years ago, the Spanish Government decided to change the conditions and requirements set for vulnerable people to receive a discount in their electricity bill. This subsequently led to the introduction of a newly developed software – created by the public administration – that electricity companies would need to use to determine whether a citizen is entitled to the bill reduction. This software, called BOSCO, was based on a regulation that defined the specific prerequisites to obtain the discount – prerequisites that were slightly different to the previously existing ones. These changes, along with the new complexity brought by the use of BOSCO and other bureaucracy issues, resulted in a huge number of people that ended up not requesting the grant they were entitled to. (Spanish version of this post, here).
Upon this happening, Civio, a Spanish organisation with the purpose of monitoring public bodies and powers, in collaboration with the National Markets and Competition Commission, created and launched a free application that would allow people to assess whether they were eligible for the grant, as well as help them with the bureaucratic process. However, through this app, they ended up discovering that BOSCO was apparently missassesing eligibility, since people that were entitled to the grant according to the requisites established by the regulation and according to Civio’s app were rejected by BOSCO.
The BOSCO case
In order to enlighten the issue, Civio requested the Ministry, and later the Council of Transparency, access to BOSCO’s source code. However, the Ministry, as well as the Council of Transparency, rejected their request and refused to give access to the code, so Civio filed a lawsuit against that decision. Based on a number of factors, the lawsuit was dismissed and Civio was rejected access to the source code of BOSCO. The judge upheld four main arguments: firstly, that BOSCO is a mere part of the administrative proceedings, so its outcome does not constitute an administrative decision. Secondly, that the non-disclosure of the code makes the application more secure, and since the code is not reusable, as it too specific, it should not be disclosed to avoid endangering security. Thirdly, that the algorithm has already been audited and that the outcome can be cross-checked with the norm to verify whether someone fulfils the requirements and is entitled to the grant. And fourthly, that public security may be threatened by disclosing the code.
Subsequently, Civio challenged this judgement. In the meantime, the Council of Transparency undid itself and sided with Civio, but the Ministry still opposed to Civio’s responding lawsuit. On May 2024, this last lawsuit was also dismissed on similar grounds.
every time I read ‘BOSCO’, mi brain can only picture Wall-E
Lessons from BOSCO – And further
Not only the Council of Transparency sided later with Civio, but also the Secretary of Digitalization and Artificial Intelligence did by declaring to be very unsatisfied with the BOSCO case, upholding the main arguments defended by Civio. The Ombudsman Office has called for further safeguards on the whole process, and the Government is currently studying a way to improve it.
Also, it is to be noted that the general position of the Spanish Government and other administrations on algorithmic governance is clear in this regard. In 2021, the Spanish Government presented the Charter of Digital Rights, that enshrines a series of rights when citizens face the use of artificial intelligence systems.
Additionally, initiatives like a Public Algorithms Register in Barcelona and shortly in Valencia show that steps are being taken towards transparency and good governance. For the private sector, the Parliament passed a law that regulates algorithmic transparency in the employment field. This norm gives workers the right to be informed about the parameters, rules and instructions via which algorithms or AI systems impact their working conditions and determine access to employment.
Hence, the BOSCO case looks more like an isolated case of malfunctioning rather than anything else. However, it is the whole point of norms and governance to create a framework to prevent such cases, or to offer solutions when they happen. This article will try to illustrate why good algorithmic governance is important, especially focusing on the public sector, how it is currently regulated, and how we can use these norms to obtain redress when something like this happens, as well as their limitations and proposals to address them.
A constitutional approach – What’s at stake?
The rule of law is enshrined in Article 2 of the Treaty on European Union as one of the common values for all Member States. Under the rule of law, all public powers shall act within the constraints set out by law, in accordance with the values of democracy and fundamental rights. The World Justice Project lays down a series of principles that build a rule of law system: accountability, meaning that the government as well as private actors are accountable under the law; just law, meaning that the law must be clear, publicised, and stable and must be applied evenly, and that it must ensure human rights as well as property, contract, and procedural rights; open government, meaning that the processes by which the law is adopted, administered, adjudicated, and enforced are accessible, fair, and efficient; and accessible and impartial justice, meaning that justice is delivered timely by competent, ethical, and independent representatives and neutrals who are accessible, have adequate resources, and reflect the makeup of the communities they serve.
In this sense, in Spain the Constitution enshrines the right of access to the law in the form of the principle of publicity of the norms. According to this principle, the people have the right to know of the existence and content of every norm that governs them. Following the same reasoning, another backbone of the rule of law is the right of people to know the reasoning of legal decisions that affect them, which is also generally applied in administrative law as the principle of motivation of administrative acts.
The two above mentioned principles can easily get impaired when code becomes law, if code is not disclosed. When a law is directly applied by an AI system, the system itself will become a self-enforceable law. In this case, the source code will be what will determine how the system works and how it reaches a decision. This means that the source code will basically be the text of the law, that determines how the law is applied. Hence, if we allow source code to become law by the way of automating such law, it should become law to a full extent. Among other implications, this shall also mean that the source code, as law, must be accessible to the people affected by it.
This concept, coined as lex informatica by Joel Reidenberg, was subsequently popularised as “code is law” by Lawrence Lessig. The code-ification of the law poses a series of inherent risks, the most obvious one being its lack of capacity to interpret any norm or to adapt to a case-by-case basis: the code just rigidly executes what is written, with no room to interpretability.
A second relevant risk, that is not inherent to code but to the way we behave around code, is the lack of transparency. Nowadays, as seen in the case of BOSCO and many others, there is a reluctance to give access to the source code of a system. This lack of visibility into the code, as explained, makes it impossible to audit the system in order to know whether it functions as it should, i.e., it makes it impossible to get access to the text of a law that is enforcing itself.
A legal system is more than the sum of every legal text that composes it. Laws include introductory statements that help understand why they are necessary and how they need to be interpreted; they include principles to guide how they need to be applied, and include clauses and exceptions open to interpretability. Laws are later interpreted by judges, courts, and administrative bodies, on a case-by-case basis and taking all the latter into account. In conclusion: laws need to be able to adapt to a complex reality.
A practical approach – Applicable legislation
Given the nature of AI systems, two main regulations are applicable to this case: the General Data Protection Regulation (GDPR), and the Artificial Intelligence Act (AIA). Other pieces of legislation relating to human rights and AI will also be taken into account.
Data protection legislation
BOSCO, as most AI systems, processes personal data. This subjects it to the General Data Protection Regulation. This norm enshrines in Article 22 a general right not to be subject to automated decision making (ADM), when such a decision legally (or similarly) affects someone. When exceptions are met and ADM happens, it also grants the right to obtain human intervention, to express their point of view, and to contest the decision.
Although this Article provides for some exceptions to this prohibition, none of them apply. One of the exceptions can be applied when ADM is authorised by the law. However, in order to be able to apply it, suitable measures to safeguard the data subject’s rights and freedoms and legitimate interests shall be laid down in the law. This is something that does not happen in this case, since it is not specifically authorised by a state law and appropriate safeguards have not been established, as reasoned.
In the Netherlands, the SyRI case can help us understand certain implications and shed some light on ADM in the public sector in similar cases. The Dutch Government’s System Risk Indication (SyRI) algorithm was an AI system for fraud detection targeted at neighbourhoods hosting poor or minority groups in the country. The system built risk profiles of individuals to detect various forms of fraud, including social benefits, allowances, and taxes fraud.
This case was investigated both by the District Court of The Hague and the Dutch Data Protection Authority (Dutch DPA). One relevant finding from the Court is that the system had a significant effect on the private life of the affected persons, breaching Article 8 of the European Convention on Human Rights (right to respect for private and family life). This implies that cases like this one shall fall under Article 22’s “legal effect”.
Another relevant finding by the Dutch DPA determined that, irrespectively of whether Article 22 GDPR is applied, principles from Article 5 GDPR must be respected nonetheless. For what is relevant to this case, this includes the transparency and accountability principles.
On another note, the ADM prohibition requires for the decision to be solely automated. However, case law accepts some degree of human involvement for it to continue to fall under Article 22. According to guidance from the European Data Protection Board (EDPB), not all forms of human involvement in a decision-making process rule out the application of Article 22, as mere token gestures taken by humans are not enough to set aside the ADM prohibition. In the case of BOSCO, even if the judgement declares that the decision is formally made by an administrative body and not by the system (since legally speaking an AI system cannot officially make an administrative decision), human involvement in the decision itself is de facto nowhere to be found. Hence, it shall be considered solely automated decision making, and Article 22 shall apply.
The GDPR also grants transparency rights to people affected by ADM in its Article 15. Those rights include the right to know about the existence of such decisions, and to obtain meaningful information about the logic involved, as well as the significance and the envisaged consequences of such processing for the data subject.
Artificial intelligence legislation
The Artificial Intelligence Act was only a draft when the BOSCO case started, and most of it has not yet fully entered into force to this day. However, looking to the future, it is helpful to understand how this regulation may affect similar cases one day.
The AI Act regulates the development, deployment and use of artificial intelligence systems in the European Union. It uses a risk-based approach with four levels of risk: unacceptable risk, high risk, limited risk, and minimal risk.
High risk AI systems (HRAIS) include, among others, those that are used for evaluating (for or as a public authority) whether essential public assistance benefits and services are or continue to be available to a particular person. Therefore, BOSCO, aimed at evaluating whether citizens are entitled to an electricity grant due to vulnerability reasons, must be considered a HRAIS.
These systems are subject to several obligations, most of them related to transparency and accountability, but also to robustness, accuracy and resiliency. Interestingly, one of them is that the AIA envisages that HRAIS shall have a conformity assessment carried out by an appropriate third party, what means that the legislator intends for these high-risk systems to be independently audited.Similarly, Recital 73 also requires high-risk systems to be designed so that «natural persons can oversee their functioning, ensure that they are used as intended and that their impacts are addressed over the system’s lifecycle.». The sentence “used as intended” is of great importance to this case, since this is precisely the key of what is happening to the system: it is not being used as intended because it is malfunctioning.
Additionally, the AIA lays down another obligation that establishes that affected persons shall be informed when an HRAIS is used for decisions concerning them, even if the system is only used in a supporting capacity rather than making the full decision automatedly (which goes further than Article 22 GDPR on ADM). In such a case, the provider is obliged to explain what the key elements of the decision were.
However, contrary to the GDPR, there is no individual right to lodge a complaint with a supervisory authority in the AI Act. Hence, individuals and civil organisations lack any direct redress mechanisms that can be used to enforce their rights or to start a procedure with the supervisory authorities when non compliance happens.
It is true that Article 67, for example, grants supervisory authorities the power to take any appropriate measures to bring operators into compliance where there is a risk to health or safety, or fundamental rights or public interest are threatened. However, everything is left to the authorities in charge, that will need to surveil the whole market by themselves. This means that, ideally, they would need to be able to be fully aware of every problem and to initiate every necessary actions own volition, which is, in practice, hardly impossible, since it will be extremely difficult to have the capacity to continuously monitor the whole market in order to notice every potential non-compliance case, given the size of the market and the current scale of AI use.
All this will leave authorities in a delicate position with respect to market actors, both private and public, as authorities will never have enough resources to monitor the whole market, and it will also leave individuals unprotected, not having any possibility for direct redress, as well as civil society, that will lack any resources to directly face problematic AI uses. Hence, this will create an imbalance of power that will favour AI deployers and providers and will difficult enforcement.
On the other hand, regarding internal coherence of the regulation, it equally makes no sense that individuals are given certain rights (like transparency rights) while they are not given the power to enforce them.
Apart from the European Union, other countries and jurisdictions are creating legislation on Artificial Intelligence and a great part of it is focusing heavily in transparency and accountability, as the proposed Transparent Automated Governance Act or Algorithmic Accountability Act in the US, or the Directive on Automated Decision-Making in Canada.
Following the same principles, the European Union, the US, and the UK have signed the first “legally binding” treaty on AI, which aims at ensuring that AI use aligns with “human rights, democracy and the rule of law”. The treaty lays down as fundamental principles, among others, transparency and oversight as well as accountability and responsibility.
This treaty also establishes the need for appropriate remedies, procedural rights and safeguards, that include the need for documentation, transparency practices that allow people to challenge any decisions made by the system, the possibility of lodging a complain with the competent authorities, effective procedural guarantees, and notice when one is interacting with an AI system.
Charter of Digital Rights
In 2021, the Spanish Government presented the Charter of Digital Rights, that enshrines a series of principles to be observed when dealing with artificial intelligence, including transparency, auditability, explicability, traceability, and accessibility. However, as happens in the AI Act, this Chart does not lay down any obligations that are directly enforceable nor includes any direct redress mechanisms, although it demands that people interacting with AI systems have the right to request human intervention and supervision, the right to challenge automated decisions, and the right to be assisted by a human at the citizen’s request.
oh, justice
Applying our principles
As seen through our analysis of applicable legislation, there are a series of principles that are repeated throughout most of the norms: transparency; accountability; human in the loop; and rights and redress mechanisms.
Transparency
The transparency principle, generally speaking, aims at facilitating the task of holding the representatives of the people, and institutions and other actors accountable. Yet it is in itself a backbone of the rule of law, as the legislative process, the text of the norms, and the decisions made in interpretation of the norms need to be carried out in a transparent manner. Transparency is a prerequisite to the rule of law as well as necessary for accountability.
However important transparency might be, traditional transparency on its own has proven not to be enough. Transparency cannot be the only solution in all cases but also needs for other measures to complement it. For instance, the transparency rights enshrined by the GDPR and the AIA are clearly not enough in the case of BOSCO. Here, both rights relate to the logic involved and the reasoning behind the decisions made by the algorithm. Yet the logic and the reasoning of BOSCO are already known to us, but they have still not been enough.
With BOSCO, there is no complexity whatsoever in the system’s decision making: the software only needs to check whether very simple requirements are met or not. There is no other value or reasoning behind the decision.
Interestingly, in AG Opinion C-203/22, the Advocate General considers that, from a GDPR perspective, ‘meaningful information about the logic involved’ must enable the data subject to exercise the rights guaranteed to them. That presupposes that the information, apart from being concise, easily accessible and easy to understand, must be sufficiently complete and contextualised to enable that person to verify its accuracy and whether there is an objectively verifiable consistency and causal link between, on the one hand, the method and criteria used and, on the other, the result arrived at by the automated decision.
In the case of BOSCO, this means that the information should be enough to determine whether the method and criteria used to arrive to the decision are consistent.
Additionally, the AG takes into account the existence of rights of other parties (third-party personal data and trades secrets) when dealing with delicate parts of the software or the source code itself. Where these rights may be affected, the AG offers a solution: that the necessary information is given to an authority or court to be examined, instead of giving it to the affected party. This, interpreted sensu contrario, leads to a reasonable conclusion: when such third party rights do not exist (i.e. there are no trade secrets involved nor third-party data), there is no reason not to disclose any information necessary to enable the person to verify the accuracy of the decision. In this case, there is no reason not to disclose BOSCO’s source code.
In this sense, the AI Act also grants supervisory authorities the power to request the source code of high-risk AI systems in certain cases, meaning that the source code is not meant to be “sacrosanct”.
It is true that the Court offered, in its judgement against Civio, cybersecurity reasons against the disclosure of the code. Yet this reasoning sounds like a made-up excuse rather than anything else. It is a completely outdated argument that does not hold itself; security through obscurity has long been proven to be an inadequate way of protecting software from vulnerability. Even the American National Security Agency (NSA) has an open-source project where it provides free access to the code of many of their own-developed software and tools. One would rather believe that the NSA would not provide such code if there was any risk.
But we do have closer examples without needing to cross the Atlantic: the Spanish Government has released the source code of own-developed apps for public scrutiny and auditing. The COVID tracing app’s source code was revealed due to transparency reasons, as was also done in many other European countries. If this meant a vulnerability, the disclosure of the source code of such app could risk unauthorised disclosure of health data of millions of people. However, the Government confirmed the app to be perfectly secure and privacy-friendly, while stating that technology should be put at the service of the people. Why would be the reasoning any different in the case of BOSCO or other AI systems created by or for the public sector?
Another of the main arguments against algorithmic transparency have always been intellectual property rights and the rationale behind the existence of such rights, since intellectual property seeks to protect, among others, the investment and effort used for an intellectual creation. However, when such creation is paid for or directly made by the public administration, the reasoning above makes no sense, as the investment is originally made in the interest of the citizenry and seeks no economic profit, quite the opposite to the private company.
In such a case, the public administration shall have no other interest than the people’s interest. Therefore, preventing the reuse of the source code lacks any sense, since there is no profit motivation, while the disclosure of the code may be an asset for future projects. Also, preventing the code from being audited would only lead to hindering potential improvement and error detection. “Public money, public code”.
Accountability
The principle of accountability means to be a way that allows for actors to be scrutinised and held accountable for their actions. In the realm of the highly complex and constantly evolving landscape of artificial intelligence and the impact in our lives and in society in general, accountability acquires more importance than ever.
In this sense, the AI Act requires from high-risk AI systems an elevated level of accountability. Given the Act’s risk-based approach to AI regulation, it imposes a gradual scheme of requirements and obligations depending on the level of risk posed by the system to health, safety and fundamental rights. Hence, actors are given the main responsibility for good governance and are obliged to produce an array of relevant internal knowledge and documentation, as well as to create safe processes, and to be externally audited.
Accountability, as anointed by Article 5(2) GDPR, is also one of the GDPR core principles; principles that shall be applicable to any system that processes personal data, including BOSCO, as remarked above. And once again, transparency, always linked to accountability, is another of the data protection principles to be applied, as well as lawfulness, and fairness.
The need for fair governance in the public sector becomes more and more important the more AI systems become able to affect our fundamental rights. This implies that putting special emphasis in transparency and accountability is needed. And accountability requires that we are able to scrutinise them and hold them accountable. When talking about AI systems, the administration shall make available any means for the citizenry to inspect the use of such systems.
This can be done in several ways, one of them being the necessity for independent and external auditing of the source code of public AI systems, as well as making documentation publicly available, including the source code where possible, creating a public inventory of such systems, and establishing solid supervision processes.
However, relying on accountability without supervision makes no sense. If actors know they will never be held accountable, the whole point of demanding accountability gets lost.
Human in the loop
“Human in the loop” (HITL) is one of the most recurring solutions when talking about safeguards for AI systems. The right to request human intervention, the right not to be subject to decisions made solely by a machine, or the right to challenge a decision made by an AI system appear again and again as ways of minimising the risks that these systems pose to people’s rights and liberties.
However, as seen, this solution presents some limitations. Mainly, that HITL is, in many cases, mostly an a posteriori safeguard. When talking about ADM or non-complex tasks or systems, no human will normally be involved in the process; otherwise, if a human was needed, the whole point of automatising certain actions would be missed. Happening a posteriori means that 1) the affected person will be already impacted by the AI system, possibly having their rights affected, and 2) that many affected persons may accept the decision made by the system without contemplating (or without being able to understand) the possibility of wrongness, so they may not even notice that their rights have been affected.
However, in the case of BOSCO, the algorithm is so simple that having a human in the loop would make no sense. Adding human review by default would mean that there would not be a need for using BOSCO, since reviewing BOSCO’s output or reviewing a form with a few ticks would mean the same amount of time and effort to the person in charge; hence rendering BOSCO useless.
Rights and redress
Redress mechanisms are indispensable where rights are granted; as coherent as it is, there is no better way to ensure that rights are respected than giving individuals an appropriate way to enforce them.
However, it is equally important not to place the whole weight of enforcement on individuals, since most persons usually don’t have the knowledge nor the capacity to engage in legal procedures. This is why the action of civil society is equally important, and that’s why in regulations such as the GDPR, civil organisations are also given the capacity to file complaints of behalf of individuals. Several of the most important GDPR enforcement actions come from civil organisations. An action by La Quadrature du Net resulted in the biggest fine under the GDPR (a 746M € fine against Amazon), while noyb managed to get the European Data Protection Board to initiate a taskforce to deal with several complaints regarding to the use of cookies, which led to a more harmonised application of the law. They also led the Irish Data Protection Authority to a key decision on Meta’s business model. Following an action by the Irish Council for Civil Liberties (ICCL), the European Commission will start regularly checking the progress of all “large-scale” GDPR cases across the EU.
As seen, the good functioning of individual and civil procedural rights has been proven of great importance; for this reason, the European Commission has a project to adopt new rules to ensure stronger enforcement of the GDPR in cross-border cases. As the saying goes, “laws without enforcement are just good advice”.
Conclusions
All the principles referred to above are inextricably linked together. It is the sum of them what will ensure the good governance of artificial intelligence and the respect for the human rights of those impacted by AI.
As remarked by Virginia Eubanks, an algorithm can never be a way of releasing regulators and decision-makers from human consequences of political decisions. Particularly, when those decisions are not even carried out as intended by the algorithm, as proven and explained by Civio. The fact that it is not the norm that has shaped the malfunctioning, but the software itself, reveals the compelling need of having access to the source code of BOSCO or any other algorithm impacting fundamental rights.
It is necessary for the civil society to have the capacity to examine machines that are making decisions or directly determining the outcome of a decision with a relevant effect; otherwise, hindering the rule of law would be as easy as automating every significant decision, so there would not be means to scrutinise them.
Citizens are already given certain rights with regard to the use of AI systems that affect them. However, such rights have been proven not to be enough, or that they are not being fairly applied, at least. The existing framework lays down several transparency and accountability obligations that complement each other, as well as principles that should have prevented, in theory, what has happened from happening. Yet such principles have indeed not prevented the system (as in “AI system”, in “legal system”, and in “public administration system”) from malfunctioning. Partly, because the whole weight of a good and fair governance cannot be placed solely on the citizens. Good and fair governance must come from within the administration, and the administration shall make available to the citizenry and civil actors every way possible to scrutinise them and hold them accountable.
the world if AI systems governance was perfect