Sam Bennett explores how AI generated medical data is being used to fill data gaps for marginalised groups and uses queer theory to unpack the problems with using AI generated data.
Discussions of the ‘datafication’ of healthcare are everywhere; in the news, in our experiences with healthcare providers from triage to treatment, and in Westminster debates about Big Tech investment (with recent news that patients in the English NHS cannot opt out of Palantir’s data access!). But this process of enhancing (or even replacing) our health systems requires access to ever-growing pools of data. Missing and incomplete data access complicates this picture, because machine learning algorithms depend on data to produce predictions. However this incompleteness does not affect everyone equally; certain groups are under-represented or missing in datasets. This might be a result of bias in tool design, systemic inequities or even reflect a conscious decision of certain groups to not engage with distrusted providers/ systems. Either way, machine learning requires data in order to produce accurate inferences or outputs, and AI generated Synthetic Data is being marketed as a way of dealing with this issue.
In spaces developing data-driven methods, where I have worked and advised, I have often heard that advocacy for marginalised groups means finding better ways to collect data – that to be counted is to be heard. Any “data gaps” tend to be framed as the enemy of progress: if the state, or other actors, don’t have data on us, they can’t serve us. Kevin Guyan terms this tension between representation and agency “the rainbow trap”, highlighting how to be represented you must be categorised, but to be categorised is to be misrepresented, particularly if you are part of a marginalised group typically defined by being different from the norm, such as LGBT+ people (Guyan 2015). I find this a useful way to think about tensions from how synthetic data is developed and used in healthcare settings. By thinking through a couple of examples of common issues in synthetic data generation, I ask: what does a synthetic [rainbow] trap look like? Are there any realistic ways to avoid falling into this trap? And…
What is synthetic data and how is it used in healthcare?
But what do I even mean by synthetic data? Synthetic healthcare data is artificially generated data that mathematically mimics the statistical properties of real clinical datasets without containing anyidentifiable patient information – essentially, it looks close enough to be usable, without fully replicating the original data. And, as you might be thinking, ‘close enough’ to real is quite subjective and differs based on the sort of context data is being used in. Traditional datasets try to translate real-world information into a corresponding number. For example you might want to know how many LGBT people report certain health symptoms to create an indication of how common said symptoms are among this population. With synthetic data, instead of pulling a real patient’s chart (which contains their actual age, genomic data, medical history), a machine learning model analyses thousands of datapoints, models the complex relationships between variables (e.g., “patients with condition X often present with symptom Y and are prescribed drug Z”), and generates outputs based on these correlations.
The Artifacts of Synthetic Data: Re-creating the Trap
So, generating synthetic data involves using algorithms trained on existing data to predict what “realistic” data looks like. But as this so-called “realistic data” has been created from selectively recorded categories and unequal representation, this process leaves behind distinct algorithmic artifacts (byproducts of the generation process) that carry deep implications for how marginalised groups are represented.
Algorithms learn from patterns, and when asked to synthesise data representing a marginalised group, the model will naturally lean on the most statistically dominant features of that group in its training set. Instead of capturing the vast and messy reality of queer or marginalised lives, synthetic data risks generating flattened caricatures. This creates a stereotyped version of marginalised identity, reinforcing rigid binaries and norms, rather than dismantling them as much of current data activism aims to do (D’Ignazio and Doğan 2025).
The Erasure of Intersectionality and Lived Experience
Intersectionality is inherently complex, and people with compounding intersections are rendered “low density” in the make-up of many datasets, which are shaped by existing structural inequality. The further an individual’s intersecting identities sit from the norm of the dataset, the more they are considered unusual, an outlier. The creators of synthetic data typically smooth out outliers to maintain a balance between ‘realism’ and data privacy (the more intersecting marginalisations, the more identifiable you are even in a ‘fake’ dataset). Plus, there’s increasing evidence of data collapse, where AI models trained on synthetic data become increasingly less diverse, regardless of intention–with the model itself erasing the margins of the distribution, that is, the outliers (Shumailov et al. 2024). In doing so, they erase the very intersections that define marginalised experiences, replacing rich diversity with a homogenised, “average” approximation of the norm.
Perhaps the deepest distortion of synthetic data is the complete detachment of data from actual lived human struggle. After all, Guyan’s critique of data emphasises that data is a social relation. So substituting marginalised voices with synthetic proxies risks creating a world where policy and technology are designed around simulations of marginalised people, rather than actual human beings. Taking an approach where you can just ‘re-add’ certain datapoints when deemed necessary treats facets of complex identities as modular and detachable rather than fundamentally co-shaping each other. Furthermore, this allows institutions to claim they are “inclusive” without ever having to engage with, fund, or yield power to real communities. When creating and challenging synthetic data, we should consider the implications of the ‘artifacts’ (unintended effects, e.g. distortions) of synthetic data – and the worldbuilding implications of curating these datasets (Bennett, Catanzariti, Emah and Garforth, forthcoming).
Escaping the Trap: A Path Forward
Using synthetic data to plug representation gaps in medical research requires critical approaches to resolving the issues outlined above and to ultimately avoid creating a synthetic rainbow trap. I share a few suggestions below, (although these are very much a starting point and this piece is intended as a conversation starter):
- Co-designing Synthesis: Marginalised communities should have agency over how conditions are represented in this new generative world. This means being included in conversations shaping synthetic medical data, particularly in focus groups and advisory boards. Discussing how do we make this data and should we even be using this data? What variables matter? What relationships between data points must be preserved, and which are harmful stereotypes?
- Embracing the “Right to be Forgotten” – and the right to never be represented at all!: Equally, sometimes the best way to avoid the rainbow trap is to accept that some things should not be modeled. Refusal (deliberate invisibility) is a valid political stance – but how do we go about it in an age where data gaps are just seen as a sign to synthesise more data?
Synthetic data promises a shortcut to data justice, offering representation without the inconvenience of navigating complex data protection agreements, providing mechanisms to alter datasets to reflect for specific needs and contexts (e.g. do you have too little information about group x? Just generate more!). But to truly understand the implications for queer representation and agency we must look closely at the artifacts of synthetic datasets and the models which generate them. If we rely on algorithms to generate our representation, we risk trapping ourselves in a hall of mirrors, gazing at synthetic reflections of who machines think we are, while real-world marginalised communities remain ignored.
What are your thoughts?
Do you think synthetic data can be a tool for data justice, or is it just the rainbow trap dressed up in newer, shinier code? Let’s discuss in the comments below.
About the author
Sam Bennett is an Assistant Professor in Durham University’s Department of Geography, and a UKRI AI Metascience Fellow, working on the project ‘Synthetic Metascience: Tracing Artificial Intelligence-generated epistemic shifts in research practice and culture”. Sam is also co-lead of the AI & Medical Humanities Research theme at Durham’s Institute for Medical Humanities. He is interested in the politics, practices and history of artificial intelligence, particularly how AI and synthetic data are shaping knowledge production in health and medicine.
References
Bennett, S., B. Catanzariti, E. Emah, and J. Garforth. Forthcoming. “Medical Synthetic Data as Alt-Real World-Building.” The Information Society.
Guyan, K. 2025. Rainbow Trap: Queer Lives, Classifications and the Dangers of Inclusion. London: Bloomsbury Publishing.
Shumailov, I., Z. Shumaylov, Y. Zhao, N. Papernot, R. Anderson, and Y. Gal. 2024. “AI Models Collapse When Trained on Recursively Generated Data.” Nature 631 (8022): 755–59.
Stevens, N., C. D’Ignazio, and A. Doğan. 2025. “Trans Data: A Research and Design Agenda from Trans Activists’ Transformative Data Science.”, Proceedings of the ACM on Human-Computer Interaction.
