Abstract
Benchmark data analysis
Abstract
The InTACT Benchmark Test is made up of 4 components, each measuring a different aspect of mentality or behaviours related to conspiracy theories. The test was carried out online in 5 countries and the answers are self-reported by the respondents.
Detailed analyses indicated that the data is highly homogenous across the countries involved. This was a sufficient basis for setting benchmark values across a range of indicators.
The benchmark values will be used to compare before-after results with regard to the conspiracy theories' deconstruction training intervention. We will also keep the test online and in data-collection mode to build a more robust dataset.
Benchmark test structure and components
The online test consisted of 4 components:
COMPONENT 1: Gullibility test
COMPONENT 2: Click-bait button
COMPONENT 3: Conspiracy Mentality Questionnaire
COMPONENT 4: Generic conspiracist beliefs scale
All components, items and scales used the original wording in English, as reported by the respective researchers, and translations into Bulgarian, Czech, Italian, Dutch/Flemish and Slovenian, produced by the researchers on the InTACT project's team. Where individual items needed to be replaced (e.g. in Component 4), we kept the text style as close to the original as possible.
COMPONENT 1: Gullibility test ¹
This component is based on the 12-item Gullibility scale (Teunisse et al., 2020). We used the original setup of the test, with a 7-point Likert-scale (from 1=strongly disagree to 7=strongly agree).
Each test item is part of one of two factors:
- Persuadability;
- Insensitivity to Untrustworthiness Cues.
COMPONENT 2: Click-bait button
This component is an experimental addition to the test and is based on an idea by [reference] who, as a control mechanism for other conspiracy-related questions, explored a test setting where subjects were asked to subscribe to a fictitious newsletter. In our case we carefully framed a statement asking the subjects to click a button if they wished to quickly access information on some sensational conspiracy theory-framed topic. We we recorded the number of clicks on a non-functional button to measure the eagerness to access such information.
COMPONENT 3: Conspiracy Mentality Questionnaire ²
This component is based on the Conspiracy Mentality Questionnaire (Bruder et al., 2013). It is a 5-item questionnaire where respondents are asked to indicate the level to which they agree with the statement in the item. The level of agreement is measured on a 10-point Likert scale (from 0% = certainly not to 100% = certain), in 10% increments.
We used the original setup of the questionnaire.
COMPONENT 4: Generic conspiracist beliefs scale ³
This part of the test is based on the generic conspiracist belief scale (Bridgerton et al., 2013) and features 5 dimensions:
- GM - belief in routine governmental malfeasance;
- MG - belief in the existence of malevolent global conspiracies;
- ET - belief in the existence and cover-up of extraterrestrials;
- CI - concerns over the unethical control of information;
- PW - belief in conspiracies infringing on people's well-being and liberty.
The original study has a list of 75 items, and there is a shortened version of 15 items (3 per dimension). We used the short version and adapted the individual items to achieve a better fit with our assessment of the predominant and shared conspiracy theories in the countries where the InTACT Benchmark test was used.
The answers are collected via a 5-point Likert scale with a qualitative label associated with each point (1: definitely not true; 2: probably not true; 3: not sure/cannot decide; 4: probably true; 5: definitely true).
References:
- Teunisse, A. K., Case, T. I., Fitness, J., & Sweller, N. (2020). I Should Have Known Better: Development of a Self-Report Measure of Gullibility. Personality and Social Psychology Bulletin, 46(3), 408–423. https://doi.org/10.1177/0146167219858641
- Bruder, Martin & Haffke, Peter & Neave, Nick & Nouripanah, Nina & Imhoff, Roland. (2013). Measuring Individual Differences in Generic Beliefs in Conspiracy Theories Across Cultures: Conspiracy Mentality Questionnaire. Frontiers in Psychology. 4. 225. 10.3389/fpsyg.2013.00225. https://www.researchgate.net/publication/236615234
- Brotherton R, French CC and Pickering AD (2013) Measuring belief in conspiracy theories: the generic conspiracist beliefs scale. Front. Psychol. 4:279. doi: 10.3389/fpsyg.2013.00279. https://www.frontiersin.org/articles/10.3389/fpsyg.2013.00279
Dataset characteristics
The InTACT Benchmark test was delivered online between Sep 15 and Dec 15, 2022. We collected a total of 204 responses (Bulgaria 38, Belgium 42, Czechia 58, Italy 37, Slovenia 29). The dataset consists of 122 female and 82 male respondents.
The average age of the respondents was 26.5 years. When asked whether they or a close family or friend has been a victim of fraud, 44.6% gave a positive answer. We observed significant differences by country on this item: BG 60.5%, BE 38.1%, CZ 36.2%, IT 56.8%, SI 34.5%.
Analysis
Aggregated data and cross-component analyses
Covariance analysis across the elements of all 4 components indicates positive relationships.
COMPONENT 1: Gullibility test
ANOVA single-factor analysis indicated no significant differences in the datasets by country. Both F-test and p-value indicate insignificant variance. We proceeded to aggregate the country datasets into one. Correlation analysis between the two calculated indicators (Persuadability and Insensitivity to Untrustworthiness) indicated r=0.6, a moderately strong correlation. ANOVA single-factor was applied to gender-categorised results and no significant variance was detected in results based on gender.
The distribution for Insensitivity to Untrustworthiness has a mean=17.31, median=17, mode=12. The distribution for Persuadability has a mean=16.21, median=15, mode=12.
In an interesting finding, when testing for previous exposure to fraud, we discovered no significant differences in either Persuadability or Insensitivity to Untrustworthiness).
COMPONENT 2: Click-bait button
This section of the test is subject to exploratory analysis. We will keep collecting data but will not set a benchmark value.
COMPONENT 3: Conspiracy Mentality Questionnaire
The histogram of the cumulative field (average of all 5 items) indicated normal distribution, with slight skewness (-0.37) and kurtosis (0.07).
COMPONENT 4: Generic conspiracist beliefs scale
Correlation analysis indicated very strong to strong correlation between MG and GI (0.80), MG and PW (0.79), CI and PW (0.77). We found negligible correlation (r=0.02-0.12) between Component 4 groups (GM, MG, ET, CI and PW) and Component 3 groups (Persuadability and Insensitivity to Untrustworthiness).
Benchmark indicators and values
Benchmark values per component
COMPONENT 1: Gullibility test
Persuadability benchmark: 16.21 ± 0.44 SE
Insensitivity to Untrustworthiness benchmark: 17.31 ± 0.38 SE
COMPONENT 2: Click-bait button
We are not going to set benchmarks for this component, as it is experimental. We will be monitoring the results as they come in. Benchmark data gives a mean of 1.38 and SE of 0.20.
COMPONENT 3: Conspiracy Mentality Questionnaire
CMQ aggregate average benchmark: 55.78 ± 1.48 SE
COMPONENT 4: Generic conspiracist beliefs scale
GM benchmark: 6.15 ± 0.19 SE
MG benchmark: 6.41 ± 0.22 SE
ET benchmark: 5.13 ± 0.18 SE
CI benchmark: 6.67 ± 0.20 SE
PW benchmark: 5.95 ± 0.19 SE