In the dystopian world of the TV show "Black Mirror," the episode "Nosedive" describes a world where social media ratings determine one’s socioeconomic status and access to essential services. Using a mobile application, everyone constantly rates others on a five-point scale. Those with higher scores can access to better services and exclusive clubs, while those with low scores are penalized in many ways. While this may seem like a far-fetched fiction, the reality of today may be not too distant from this portrayal.

The first example that comes to mind is China’s Social Credit System (SCS), developed between 2014 and 2020. The SCS uses artificial intelligence "to develop comprehensive data-driven structures for management around algorithms that can produce real time reward-punishment structures for social-legal-economic and other behaviors" (Larry Cata Backer, Next generation law: data-driven governance and accountability-based regulatory systems in the west, and social credit regimes in China, 2018). The SCS in reality does not actually rely on a universal score but rather on a series of blacklists and redlists managed at different levels (municipal, local, or national). Each authority can manage its own blacklist (e.g., on those who failed to pay fines or child support) and they all converge into the National Credit Information Sharing Platform. As mentioned by Kevin Werbach in his 2022 article “Orwell that ends well? Social credit as regulation for the algorithmic age”, this makes possible that "grade A taxpayers receive customs fee waivers and low-interest loans, in addition to the "home" benefits offered by the tax collection authority". Prof. Werbach however believes that western's depiction of the SCS is is exaggeratedly negative, especially in a world where governments and corporations are extensively tracking our behavior. He sees the Nosedive scenario as more resembling to the ratings system on Uber or eBay, expanded beyond the boundaries of one service.

As noted by Yuval Noah Harari, free-market capitalism and state-controlled communism can be regarded as distinct data processing systems: the former is decentralized and the latter is centralized. It shouldn't come as a surprise then that western's versions of social credit experiments are being made mainly by private corporations, especially in the financial sector. Since the 2008 financial crises, many "fintech" online lenders began experimenting new scoring model for establishing creditworthiness. These model are based on a person's "social footprint" which is revealed by elements such as his/her social circle, or shopping habits: surprisingly, it appears that buying felt pads has a positive influence on how the algorithms forecast your financial behavior.

The risk of discrimination highlighted by these researchers became painfully real in the Netherlands. In 2013, the Dutch Tax Authorities employed a self-learning algorithm to detect child care benefits fraud. The risk indicators used by the system included having low income or belonging to ethnic minorities. As a result, thousands of families were wrongly characterized as fraudsters and suffered severe consequences. This led to the Dutch Government’s resignation and a 3.7 million Euros fine on the Tax Administration from the Autoriteit Persoonsgegevens, the Dutch Data Protection Authority, for breaching several GDPR rules. In particular, the Authority found that the Tax Administration had no legal basis for processing the personal data used as risk indicators. Under the GDPR, personal data processing is allowed only if one of the legal bases listed in Article 6 applies.

In the hyper-regulated European Union, the GDPR has attempted to address these issues by introducing Article 22, which allows individuals to opt out of "automated decision making, including profiling, and obtain human intervention whenever their personal information is used to take a decision which produces a legal effect (e.g., entering into a contract with that individual). Additionally, the proposed EU AI Act aims to place serious limitations on "AI systems providing social scoring of natural persons for general purposes by public authorities." These limitations prohibit social scoring systems from leading to detrimental or unfair treatment in unrelated social contexts or based on unjustified or disproportionate criteria.