A useful discussion of differing poll results
Grant Duncan looks at some different poll results from an Ipsos survey and a Curia survey.
An IPSOS survey in 2023 asked: “How much do you agree or disagree with the following? With parental consent, transgender teenagers should be allowed to receive gender-affirming care (e.g., counseling and hormone replacement treatment).”
59% of New Zealanders strongly or somewhat agreed with that. 26% disagreed and 15% weren’t sure.
This survey was cited in a trans-friendly news item in The Post.
Around the same time, a Curia survey asked: “Some people have proposed banning puberty blockers, cross-sex hormones, and physical sex-change surgeries for children under the age of 18 who identify as transgender. Would you support or oppose this kind of ban?”
54% supported this proposed ban. 27% were opposed and 19% unsure.
This survey was cited – and commissioned – by the conservative Family First NZ.
Majorities supported two apparently contradictory statements.
Well, not exactly. The two propositions put to the public are differently worded and don’t directly contradict one another in all respects.
The IPSOS survey asks about “gender-affirming care” including counselling and one of a number of possible medical interventions. It makes the significant qualification of parental consent, but doesn’t mention medical judgement or scientific evidence.
The Curia item asks about a “ban” (for under-18s) on three kinds of medical intervention, and doesn’t mention any psychological interventions such as counselling. “Banned”, however, is clearly the opposite of “allowed”.
It looks as if one survey shows majority support for allowing hormone replacement treatment for under-18s with parental consent, while the other shows majority support for banning it outright regardless of parental consent or medical opinion. That implies, in theory, that some people (maybe around 13% or more) are able to express contradictory opinions on this topic, depending on who’s asking and how they ask. And up to 19% of people may be unsure either way.
The problem’s in the survey design. The IPSOS question uses “soft” terms such as care, affirming, counselling and consent. The Curia question uses “hard” medical terminology, such as blockers, hormones and surgery. The language used gives differing impressions.
Different language can and will get different results. This is why it is critical to always report the exact question asked. It is not that any set of questions is biased, just that they can reflect different things.
The Ipsos survey mentions parental consent. The Curia one does not. I don’t know if parental consent would be needed for such treatment, but it is far to say assuming that would impact the result.
Likewise as Grant Duncan points out, soft vs hard terms can affect the result. Generally I prefer specifics.
And then there’s “priming”: our response to one thing will be affected by how we responded to others, especially when closely connected in time.
The question that appeared to precede the one mentioned above in the IPSOS survey was: “How much do you agree or disagree with the following? Transgender people should be protected from discrimination in employment, housing, and access to businesses such as restaurants and stores”. 84% of New Zealanders agreed with that one.
The preceding question in the Curia survey was: “Do you believe that primary age children should be taught that they can choose their ‘gender’ and that it can be changed through hormone treatment and surgery if they want it to be?” And 76% said “no” to that.
The IPSOS survey led with a question about rights (to be free from discrimination) that all respondents hold, and this set a more positive frame for the next question.
The Curia survey, in contrast, had primed respondents with a question that evoked protection of “primary age children” from premature exposure to delicate and complex issues including sex-change surgery. It was more likely to put the respondent (especially if they’re a parent) in a frame of mind to disapprove of medical intervention in the next item.
There’s no perfect survey that’s devoid of priming effects. Indeed, this very subject-matter may be too complex and highly charged to be fit for simple survey questions on a five-point scale.
I would not use the term priming, as that suggests you ask a question purely to influence later questions. I won’t do that. In fact ethically you need to publish all questions that could significantly influence the response to a later question.
The issue for pollsters is clients often want to poll on half a dozen different questions on an issue. You can’t do them in six different polls, so you have to ask them in one poll. This is not to prime people, but to find out where support lies for various propositions.
Pollsters do look at question order when questions can influence each other. If there is one question that is obviously most important, then try to ask that first. But sometimes all questions are equally important.
Again it is important that reports include all relevant questions asked in order, so people can judge if there is any influence.
Dr Cass recommends, for example, “extreme caution” and “a clear clinical rationale” if considering “masculinising/feminising hormones” from 16 to 18. So it could be provided as part of a research programme under multi-disciplinary medical and psychological supervision. Her review concludes with neither “ban” nor “allow”. It’s not that simple.
My personal view is the same. I do support under 18s being able to access some clinical treatments, but that the younger one is, the more cautious you should be, and the more serious the intervention, the more cautious you should be. So sex change surgery on a 14 year old is a no, but hormone treatment on a 17 year old could be a yes.
Polling can’t get into these complexities generally. What they can do is give an indication of where opinion is on high level issues. If you want more detailed consideration, then focus groups can be useful.
The post A useful discussion of differing poll results first appeared on Kiwiblog.