Use of statistics in the Scottish Government publication ‘Supporting Transgender Pupils in Schools’

1.  This blog considers the use of statistics in the Scottish Government publication ‘Supporting Transgender Pupils In Schools’. The analysis raises serious concerns on grounds of accuracy, data quality, and representativeness, and shows a lack of due diligence in relation to source material.[1] The way in which the data is presented, together with the graphics, is alarmist and risks exacerbating what may already be stressful situations for children and young people, their families, and schools. Given that the guidance is directed all schools in Scotland and deals with an extremely sensitive topic, and a vulnerable group of children, with implications beyond the school system, into the family home, we suggest that its use and interpretation of statistics should be reviewed.

2.  The Scottish Government guidance states: ‘Evidence shows that young people who have parents who are supportive of their transgender identity are more likely to have good mental health, including improved self-esteem and reduced rates of depression.’ (2021: 37). It is not clear what evidence base the Scottish Government is drawing on here regarding supportive behaviour, over which there is some dispute, or whether it has reviewed that evidence systematically. For example, in written evidence to the Women and Equalities Committee, a group of GPs and consultant psychiatrists argue against the approach advocated by the Scottish Government, and recommend ‘supportive affirmation of an individual’s experience rather than automatic ‘affirmation’ of being trans (which can, in effect, endorse medical intervention without and before medical input).’

3.  The linked source for the statement in the schools guidance is a briefing report by Travers et al. (2012) ‘Impacts of Strong Parental Support for Trans Youth‘, which is based on survey research conducted between 2009 and 2010, as part of the ‘Trans PULSE’ project. While the Trans Pulse project has generated academic papers (see here), the briefing cited in the guidance for schools is self-published and not peer reviewed.

4.  The same briefing is cited as the source for an infographic in the guidance for schools, titled ‘Why support for trans youth matters’, as reproduced below.[2]

5.  The infographic cites several statistics from the 2012 briefing, including that 4% of children with supportive parents attempted suicide, compared to 57% with non-supportive parents. This statistic is illustrated with two pills, clearly intended to signify an overdose. The infographic is based on an earlier version published by Trans Student Educational Resources (TSER),[3] which illustrates the same suicide statistic with a noose.[4]

Accuracy

6.  The infographic in the schools guidance states that the findings are ‘based on a study of 433 respondents’. This is incorrect. As the linked briefing notes, the findings are based on data from 84 respondents, aged 16 to 24 years, who had ‘socially transitioned gender (or begun to), come out to their parents’ (p.2). The age-range of the full sample cited by the Scottish Government (i.e., N=433) ranges from 16 to 77 years (Schiem and Bauer, 2014: 5).    

Representativeness and data limitations

7.  The Trans PULSE study is clearly not generalisable to the wider population of Scottish school children.

8.  As the study only collected data on the trans population it is not possible to compare the outcomes it reports against those for non-trans young people. It therefore fails to provide any context for the figures it reports.

9.  The study excluded any cases obtained in the full survey who were aged 16-24 and had reported a transgender identity, but who had not begun any type of social transition even if, it appears, they had shared their feelings with their parents. It therefore as a deliberate decision reports no information about that group, although they are at least equally relevant in this context.

10.  The Trans PULSE study uses ‘respondent-driven sampling’ (RDS), a type of snowball sampling.[5] An academic journal article using the study data explains that a ‘group of 16 well-connected trans people served as “seeds” or initial participants’, which was later supplemented by a further 22 ‘seeds’ (2011: 137). This method, which relies on network connections, is likely to over-sample those with particular shared world-views, and under-sample those who do not use advocacy groups or require their support, who may differ significantly from the respondent population. The TSER report states that its findings relate specifically to ‘the population of networked trans people in Ontario (those who know at least one other trans person’ [who themselves knew at least one other trans person, and so on]. It cannot be assumed that those who are not connected to the wider trans population in that way would report the same type of experiences.   

11.  The Trans PULSE survey gathered all its data in a single snapshot and so does not allow for causal inferences to be drawn. In a separate paper, its co-authors state that findings from the survey ‘must be interpreted cautiously’ (Rotondi and Bauer, 2011). The emphatic presentation of figures from the report by the Scottish Government as generalised statements breaches that advice.

12.  To produce statistical findings generalisable to a wider population, data has to be drawn from a random sample of a relevant population and reported with appropriate estimates for the margin of error, which will increase as the sample size falls. This study therefore does not provide any basis for making robust estimates for effects beyond the immediate group involved in the study. Although a separate report from the Trans PULSE project clarifies: ‘all statistics presented are generalizable to the population of networked trans people in Ontario (those who know at least one other trans person’ (Sheim and Bauer, 2015: 1), in the absence of random sampling even this claim is questionable. There is no basis at all for assuming its findings will predict experiences with any accuracy for Scottish school children aged fifteen years or under.

Survey methods and design

13.  The questionnaire used in the Trans PULSE survey is exceptionally long at 87 pages. This risks response fatigue and inaccurate answers, particularly for younger respondents. The question on which the analysis in the briefing is based is shown below (at page 60 of 87). The cases studied were those which had responded to this question: there is no way of telling how many of those excluded from the study would have met its criteria for inclusion, but skipped the question, introducing a further confounding factor.

14.  The briefing states that participants ‘provided information reporting how supportive their parents were of their gender identity or expression’, however what was meant by ‘supportive’ in this context was not defined, nor any examples given, and was therefore open to respondents applying widely varying interpretations to identical parental behaviour.

15.  The presentation of the questionnaire appears partisan and politicised, which risks eliciting respondent bias (for example, encouraging those with similar views, and vice-versa). This is conveyed by the inclusion of short biographies of the project co-ordinators within the main body of text, for example, one team member is described as a ‘2-Spirited Transsexual Warrior. A graduate from the school Hard Knox’ and former ‘Prostitute, Bank robber, Heroin/Cocaine addict, and thus ending up in Federal Prison for 5 years’ who has ‘obtained sex-reassignment surgery, and turned over a new leaf’ (p.21).

16.  The questionnaire also incorporates of what it terms ‘Trannytoons’ cartoons within the main body of text. These give light-hearted advice about topics such as recovery from surgery. They would not normally be regarded as appropriate to include in a research questionnaire seeking to gather unbiased data.

Data analysis

17.  The briefing cited in the guidance for schools does not provide any demographic information on the respondents. For example, we do not know the sex of the participants, their age (by year), income, or employment status. There is no assessment of how the mental health status or history of individuals reporting different outcomes may vary. No other factors are explored as potential contributory factors to the outcomes of interest. 

18.  The data in the briefing is analysed for ‘two levels of parental support’ across the 84 respondents. These are ‘strongly supportive’ (34%, n=29) and ‘not strongly supportive’ (67%, n=56).[6]  However, the ‘not strongly supportive’ group collapses three response options: those with ‘somewhat supportive’ parents (25%, n=21) and whose who with ‘not very’ or ‘not at all’ supportive parents (42% n=35). The split between the latter two response options (i.e., ‘not very’ and ‘not at all’) is not shown.

19.  In effect, this means that the results reported for the group with ‘unsupportive parents’ were based on the responses collected in Canada over a decade ago from just 56 non-randomly sampled people aged 16 to 24, of whom 21 had in fact reported their parents as being ‘somewhat supportive’. 

20.  The briefing does not explain why ‘somewhat supportive’ is bracketed with the ‘unsupportive’ responses. This muddled binary grouping, together with the very small numbers in the survey, precludes any conclusions being drawn about the association between the varying strength of parental support and outcomes.

21.  In a separate Trans PULSE presentation, using the same data but showing three separate categories for supportiveness, there are marked differences between the ‘somewhat supportive’ and ‘not very/not at all supportive’ groups, which when considered alongside the results for ‘very supportive’ substantially weaken the argument that greater parental support correlates with lower suicidal intention and other outcomes, even leaving aside all the objections above.  

22.  For example, those with ‘unsupportive’ parents reported higher life satisfaction, were slightly less likely to report depressive symptoms, and much less likely to have reported attempted suicide in the past year, compared to those with ‘somewhat supportive’ parents. These differences are shown below. Figures 1 and 3 show the data from the briefing as was reproduced in the guidance for schools, which is based on two categories, and Figures 2 and 4 show the same data, as shown in the presentation, which is split into three categories

Figure 1. Trans PULSE briefing cited by Scottish Government (life satisfaction): Binary ‘supportive’ categories

Figure 2. Trans PULSE presentation (slide 29) (life satisfaction): 3 ‘supportive’ categories

Figure 3. Trans PULSE briefing cited by Scottish Government (depression; suicide attempt in past year): Binary  ‘supportive’ categories

Figure 4. Trans PULSE presentation (slide 29: depression; suicide attempt in last year): 3 ‘supportive’ categories

23.  In response to a Times article (16 August 2021) on the use of the infographic statistics, the Scottish Government stated: ‘The infographic contained within the guidance is intended to be illustrative of the concerns which transgender young people may face. The infographic clearly contains the reference to the study as the source, and therefore is appropriate.’

24.  We think that this response is wholly inadequate and fails to acknowledge the very substantial limitations of the underpinning study and source data, which mean that it falls far short of being suitable for quoting in this guidance as a source of statistical information relevant to the Scottish school-age population, on a sensitive topic.

25.  We also believe the use of a graphic of a potential method of suicide is ill-judged in this context and reinforces the unavoidably sensationalist effect of a comparison of figures of 4% and 57% for attempted suicide in a group of school age children. 

26.  We also note the unqualified use of other survey data in the guidance. For example, it is stated that ‘63% of transgender young people experienced suicidal thoughts or behaviours’ (2021: 10), without any reference to sample sizes,[7] data collection methods or representativeness in the cited survey (Dennell et al. 2018). Respondents to this survey were also self-selected, again making it unsuitable as a basis for generalising to a wider population. We note further that Dennell et al. do not disaggregate between trans and LGBT respondents in the sample size, which makes this statistic difficult to interpret (see Figure 3). Working back, the demographic data in Appendix 3 shows that the survey secured 684 responses, of whom 486 reported their gender identity. Of these, 33.5% (n=163) identified as transgender (the demographic data is not clear, but we think this is correct). No information is provided on how many of the trans identifying young people answered the question on suicidal thoughts or behaviours. Again, this does not provide a reliable enough basis on which to make such a strong claim on something so sensitive in a government document going to all schools. 

27.  Any discussion of attempted suicide in school age children requires very careful, conscientious treatment. Any material included on this in a government document should be subject to the highest standards of due diligence. To criticise how these figures have been used is not to dismiss the need to be alert to any such risk in this group, nor to disagree that parents’ behaviour can have important effects on their children’s well-being. It is to argue for taking the greatest care possible in any official communications and guidance on this subject, that relates to any group of children.

 

---

Notes

[1] We draw here partly on analysis shared with us in confidence.

[2] The same infographic is also shown in the LGBT Youth Scotland schools guidance, ‘Supporting Transgender Young People’. The Scottish Government produced its own guidance for schools, following strong criticism of the LGBT Youth Scotland guidance, whose advice Ministers recognised as potentially unlawful. See column 19. 

[3] TSER was founded in 2011 by two sixteen-year olds, and is run by young people. 

[4] TSER state ‘We have created an infographic (http://transstudent.org/youthsupport) based on “Impacts of Strong Parental Support for Trans Youth”. Since it was released, this graphic has been on thousands of web sites and blogs and has been seen by hundreds of thousands of people’.

[5] Further information on RDS published by the Mailman School of Public Health is available here. This states: ‘The extent to which RDS-derived estimates are valid and generalizable remains a source of controversy in the peer-reviewed literature’.

[6] Percentages do not add due to rounding.

[7] We also note the guidance for schools reproduces a participant’s quote from Dennell et al. and attributes it to a child with a trans identity, when their identity is not stated in the original report.