A tale of two letters revisited

The Herald on Sunday this week reported that “more than 300 scientists are calling for the Scottish Government … to resist attempts from campaigners to change the guidance or question around sex” in next year’s census. The call was made in an open letter. The letter text was not included in the print edition but is available on the Herald’s website.  The full text is also available online separately here.

This blog argues that the reporting overstated this as an expert intervention, and that more careful scrutiny reveals it is better understood as an example of campaigning round a shared belief, rather than shared relevant expertise, by people who happen to be based in universities, mainly in North America.

We discussed a similar letter here, a little over a year ago. This previous letter was relied on by the Cabinet Secretary, Fiona Hyslop MSP, to play down concerns expressed by a large group of quantitative social scientists, all of whom had indicated their areas of expertise, about plans to guide respondents to answer the ‘sex’ question in the Scottish census based on their self-identified gender, rather than their sex as a physical fact.

Multi-signatory academic letters, the census and the Scottish Government
The letter provided no information on the signatories’ academic areas of expertise. Further investigation showed that the group worked in areas as diverse as creative writing and theology, but very few had any evident background in working with human population statistics. Despite the Minister relying on the letter in a Scottish Parliament evidence session, an FoI request revealed that the National Records of Scotland had not investigated the signatories’ relevant expertise or background.   

We argued then:

‘NRS failed to undertake basic due diligence on a letter before it was used by officials, and Ministers were presumably briefed to refer to it, to support the Scottish Government proposal, while at the same time discounting the views of professional statisticians and population data users.  Any person with access to a search engine can quickly establish the point made by the Committee convenor; that however passionately they feel about this issue, very many of the signatories come from academic disciplines which will not expose them to working with population data.’

Background to the letter reported this week
The most recent letter was prompted by recent legal proceedings brought by feminist campaign group Fair Play For Women (FPFW).  

FPFW successfully pursued a case last month against the Office of National Statistics (ONS), which manages the census for England and Wales. The case concerned how respondents should be guided to answer the ‘sex’ question, which provides respondents with two possible answers: ‘female’ or ‘male’. The ONS guidance initially allowed for self-identification (based on unspecified ‘legal documents’, such as a passport). However, following the legal challenge it conceded that under the Census Act 1920 ‘sex’ could only mean sex as recorded on a birth certificate (or a GRC where relevant) and that it did not have the powers to guide respondents to enter any other response.  More detail about the case can be found here.

The inclusion of a new, separate question gathering data on the transgender-identified population was not challenged in the court case: there is widespread support for gathering better data on the size and other characteristics of this group.

The court case related only to the England and Wales census, which took place on 21 March this year, although following its conclusion, the guidance for the separate Northern Ireland census was amended in line with the outcome. Scotland’s census had already been deferred until next year, and the Scottish authorities have commented that they are considering the implications of the court case.    

The letter reported in The Herald misunderstands partially what the case was about, wrongly assuming that the new question intended to gather data directly on the transgender population was also being challenged.  The legal challenge focussed solely on the guidance to the sex question.

The letter was produced after a philosophy PhD student based in California who is very active on social media on the wider topic of trans rights, and has directed considerably negative attention towards a U.K. based academic with whom she disagrees, tweeted that she felt “statisticians” should be more engaged in the debate, after seeing a newspaper report of the case. The request was taken up by a professor elsewhere in California, who is a laboratory-based psychologist and does not appear to use population data. 

Who are the signatories?
The letter describes its signatories as:

‘an international team of scientists with extensive expertise collecting and quantitatively analyzing high quality data from human participants. Our group includes current and former editors of scientific journals, statisticians, and analytic instructors; collectively, we have many decades of experience working with quantitative human data, survey methods, and complex datasets.’ 

It therefore appears at first sight as if it should be a relevantly expert group.

The Herald did not publish the names of the 304 signatories, but the organisers have made them available here.  As with the letter we analysed last year, they are listed without any information about their area of academic expertise, only in most cases their location and job title. 

The Herald report mentions that signatories include “academics from Edinburgh, Glasgow and Aberdeen universities” and “across the globe”. It does not explain that signatories from Scotland are a very small proportion of the total (5%, n=14) and that more than twice as many are from the U.S. (45%) than the U.K. (21%). The balance of signatories no doubt reflects the letter’s origins. By far the most represented institution is the University of California, where the organiser is based, with 24 signatories. The next largest group by country is from Australia (11%), with the remainder from a further 15 countries, including Canada (7%) – taking the proportion of signatories based in North American to over half – Argentina, Israel, Iceland, the Netherlands and Thailand. Around 25 signatories work for private organisations, including Google, a logistics consultancy, a sexual and reproductive rights organisation and the World Bank. A further 18 do not state any affiliation. Around 16% are postgraduate researchers.

Filling the information gap on the academic expertise of all the signatories would be a huge job. Instead, we looked at the fourteen based in Scotland and then at a random sample of thirty more (excluding Scottish signatories and those who provided no institutional affiliation). In each case we used an internet search to identify what academic discipline they worked in and, if the relevance of that to this subject was not immediately clear, whether we could find evidence of published work which appeared likely to involve analysing large scale population data. The judgement here was not straightforward in every case and immediate information on some cases was sparse. The figures below are therefore best seen as giving a broad indication of the mix of signatories.

Of the fourteen Scottish signatories, eleven are academic staff, two are postgraduate researchers and one is an independent research consultant. None are established quantitative social scientists. One has made limited use of descriptive population statistics in consultancy reports and one of the postgraduate researchers is a co-author of a paper involving the complex analysis of large population data sets in a health setting. Some, mainly based in psychology, have undertaken more complex quantitative analysis of experimental data which uses human subjects. The remainder are largely involved with aspects of mathematical modelling, computational science or statistical theory, but have no immediately obvious track record of detailed population data analysis.

Of the thirty randomly chosen signatories, eleven came from England or Wales. Twenty were academic staff at varying levels, eight were postgraduate researchers, one worked with a software company and one was a health service data analyst. Allowing for disciplinary boundaries being fuzzy, and depending on how a broad reading is taken, between one-quarter and one-third appeared to have published any work which might be classifiable as large scale quantitative social science or epidemiology (the study of health effects at a population level). Up to around one-half appeared to have done some other quantitative work using people, of which the largest group was psychologists using experimental or survey data with human subjects: the rest had other types of experience of working with data about people, mainly small scale, or a job title suggesting they might do so. That left just over a quarter for whom we could not immediately find any evidence or suggestion of quantitative work examining human subjects: these were in areas including computational science, bioinformatics and soil science.

It seems likely from this that most signatories of the letter are not working with large population data sets in either the social or medical sciences.  A large minority of the signatories who do have experience of working with quantitative ‘human data’, possibly the majority, seem likely to be using that for experimental or survey-based psychological testing, rather than demographic analysis.  A further large minority seem likely to have just a general expertise in statistics. 

Who is an expert?
These findings throw light on the claim that the letter’s signatories have expertise in quantitative data about human subjects or complex data sets. Where they have expertise in quantitative data about people, this often appears to be of working with experimental subjects, especially in psychology. Perhaps this explains the reference to “human participants”, not a term that normally comes up in relation to the census or other population data.  Other signatories seem to have assumed that working with any type of large data set, or in statistical theory, confers relevant expertise here.  

This neglects to recognise that working with large population datasets is an area of expertise in its own right, found mainly in the social sciences and parts of medicine. The current debate about the sex variable in the census is concerned with what data is gathered on sex at a population-level, and in particular how that will affect researchers’ ability to investigate the relationship between sex and other demographic characteristics, especially in sub-groups with a higher incidence of transgender identities.  It is not self-evident that working with people-related data in smaller scale experimental settings or using surveys to run experiments will necessarily mean someone’s expertise is relevant here.  Statistical skills, however sophisticated, need to have been applied in a relevant context. Even something as basic as survey design is a skill which applies differently for different purposes in different settings.

Our sampling may have been unlucky (although for Scotland we examined the complete list) and there may be more cases proportionately with relevant expertise among the remaining signatories. However, when anyone asserts themselves to policy makers, and by extension everyone else, as a figure of relevant authority, then the burden of proof rests on them to make clear what their expertise is. Other people should not have to go looking:  if they do, they should be able to get clearer results than we found here.

We also looked at what expertise signatories were told they needed to have and what information was collected on them (see footnote).  It is easy to see how people with no expertise in population data, or working with human subjects at all, may have assumed they could sign, if they did not check the letter text itself. Given the limited information signatories were required to submit, it is also hard to see how the organisers knew enough to be able to make the specific claim they have about the nature of the signatories’ expertise with certainty.

The letter’s underlying argument
The final paragraph of the letter states:

… we strongly oppose the misuse of scientific-sounding claims to justify the oppression of vulnerable groups. Cloaking abominable moral positions in the veneer of scientific-sounding justifications is not a new phenomenon; it has a long history including slavery, eugenics, forced sterilisation, the denial of women’s suffrage, and more. Our scientific integrity precludes us from taking part in or endorsing such pseudoscientific interests while our ethical responsibility as scientists compels us to speak out against them. We therefore strongly reject any claim that measuring lived sex would prevent the 2021 English and Welsh Census from collecting robust and accurate data about sex.

This suggests that what the signatories appear to have in common more than relevant technical expertise is a belief – we would say it is an extreme belief – that gathering data on sex in human beings as a category grounded objectively in biology rather than subjectively as “lived sex” is morally unacceptable, and belongs in a list with, for example, slavery.

What sort of letter is this?
The letter was reported by the Herald simply as an expert intervention in the census by “scientists”, including ones from Scotland. It was not made clear to readers that this is in essence a letter organised on the back of an adverse reaction to a news report about the U.K. census by Californian university-based campaigners, whose academic expertise lies in fields other than quantitative social science or epidemiology, and who have misunderstood some of what the debate here is about (the letter text reveals). The signatories appear to reflect that. None of the small number from Scotland are established academics whose research relies on being able to do complex analysis of large scale demographic data in which the results for men and women may need to be separated for any purpose.  

Does this letter matter?
The outcome of the legal case in England on the census suggests that the Scottish Government faces the same constraint as its counterparts south of the border: it is covered by the same overarching legal framework for the census as applies in England and Wales. However, the Scottish Government may try to argue that the outcome does not set a precedent it is obliged to follow, or to use its powers to amend that legal framework for Scotland only. 

The Scottish Government’s comment to The Herald on Sunday gives no clues about what weight it might place on this letter but, given recent history, it might yet be cited as a relevant expert input, or be promoted to the government or other politicians as such, to justify ignoring developments in the rest of the UK.

Though direct university representation in parliament has long vanished, academics can still sometimes command political and media attention, just by getting together and asserting they should be listened to. We suggest however that when long lists of university-based names are offered up as authorities in support of a specific policy position, policy makers (and journalists) should always make sure they are satisfied that any claims of relevant academic expertise are supported by some evidence, and not just asserted, before giving those views extra weight.

_________________________________________________________________________

Footnote

The text was shared with potential signatories using a Google document, which did not specify expertise was needed in working with data on the human population specifically.

The criterion of “expert in quantitative data, surveys and/or large datasets” could embrace people whose quantitative expertise relates to counting anything from sub-atomic particles to galaxies.

A similar formula was used in the form signatories were asked to complete, except that being “an expert” was downgraded to “experience working with”.

This document may have been all that some people who added their names saw if they were simply sent this link and did not seek sight of the letter text.

The signatories’ form included a field for “professional title” but not one for academic discipline. It also asked for the additional information below, but the organisers will have been unable to verify that a person’s expertise was specifically in data involving people, unless that was incidentally volunteered.