Methodological Toolkit

Funded by the European Media and Information Fund, the consortium of policy research institute Political Capital, Hungarian fact-checking outlet Lakmusz and media think tank Mertek Media Monitor identified, monitored, analyzed and debunked sponsored disinformation during the campaign for the 2024 European Parliament and local elections in Hungary on 9 June. This was the first ever attempt in Hungary to monitor how much money social media platforms make from promoting false and misleading information.

This methodological toolkit describes in detail the methods, techniques, and tools used in our activities. It also highlights the difficulties with the available data, other implementation issues and limitations. The goal of the toolkit is to inspire and support the work of other researchers and organizations to help reduce electoral manipulation in the future.

The summary of the activities including the main results and links to all outputs of the project “The marketplace of (false) ideas: Uncovering, analyzing, debunking and researching sponsored disinfo” are available in the summary report here.

1. Qualitative analysis of advertisements – hostile narratives

 

During the qualitative content analysis of the ads, we sought to answer the question of what kind of hostile narratives are being propagated through paid ads on Meta and Google. Who are the purveyors and sponsors of these narratives, and how much money is spent on these ads.

Definition of hostile narratives

We focused on narratives that fit the definition of what is known in the literature as "hostile narratives". This definition is broader than disinformation, but not synonymous with the term propaganda. Hostile narratives are defined by the following characteristics:

• Deception, disruption, and deliberate and intentional distortion of reality. These narratives are not necessarily based on completely unsubstantiated, false claims. They may also be based on real, true information. However, it is not the facts, true or false, that play the main role in these narratives. What is important is the way in which they are narrated. This is usually achieved by manipulating the facts, taking them out of context, highlighting them and editing them in a tendentious way.
• Exploiting fears, exaggerating threats. Major changes that have a significant impact on our daily lives, as well as concerns and fears about the potential negative consequences of societal, political, economic, social, cultural, technological and environmental transformations, provide fertile ground for the proliferation of hostile narratives. These narratives exploit societal vulnerabilities, exaggerate potential dangers and risks, and present them as imminent and existential threats, often framing them as a matter of security, physical well-being, culture, economic stability, social cohesion and even physical safety.
• Use of an enemy image. Hostile narratives link exaggerated threats to societal/political actors (institutions, organizations, groups, individuals), name them as responsible and designate them as enemies. In doing so, they either build on existing stereotypes, prejudices, fears and resentments or create them around a previously unknown actor or phenomenon.
• Propaganda is not the same as hostile narratives. The manipulative nature of some content does not necessarily make it "hostile." For example, a clickbait title may be manipulative, but this does not make it "hostile". Similarly, content that has a strong emotional impact or propagandistically pushes a particular message is not necessarily "hostile".

Retrieval of individual advertisements

For the narrative analysis, we processed individual advertisements. The results of our content analysis were presented in fortnightly reports. In each report, we processed the ads that were active in a given 14-day period. In the case of Meta, we downloaded via the Ad Library API all political ads that were shown in the two weeks under review and on which at least HUF 50,000 was spent. It is important to note that the spending associated with an ad includes the total amount spent over its entire lifetime (from initial launch to query), not just the selected two-week window. No such or other sub-period spend data is available in the Ad Library. The data was retrieved using the Radlibrary package in the R software environment. Google ads were retrieved through the Ads Transparency Center. The regularly updated database contains all political ad data for all countries in a single file. The file size is therefore very large, reaching 2.7 GB at the time of the research. Due to the large size, filtering and cleaning was also done in R. The pre-processed ads that were active in Hungary during the study period were exported to a csv file that could then be handled by standard spreadsheet software.

Processing and categorization of ads

After compiling the database of individual ads, we processed the ads one by one in the following steps:

1. Relevance check: The first step in examining ads is to determine whether the advertised content is relevant to the research. Only content that is deemed relevant needs to be investigated further. For the purposes of this research, ads that were part of the European Parliament or local election campaign were considered relevant, excluding, for example, ads that were social or that encouraged people to follow the politician's website but did not carry any other political message.
2. Identifying content type: During the research, when categorizing relevant ads, we noted whether the advertised content contained video, text, images or links. Videos were subject to separate content analysis later in the research (see Chapter 5 Qualitative Analysis of Megafon Videos).
3. Repetition: In many cases, there were multiple versions of the same ad, with the same creative content, possibly targeting different audiences and at different costs. For the purposes of content analysis, these could be considered the same, so we flagged the repetitions so that they only needed to be categorized once.
4. Ad analysis: Viewing the video, reading the post and the linked content is the most important step in categorizing the ad. In this phase, we identified the theme of the ad, the message being conveyed, the political narrative, and looked for claims that could be fact-checked later. In the course of the analysis, particular attention was also paid to the collection of expressions ("stamps") that could be used to illustrate the process of enemy formation in the simplest way.
   1. Identifying the theme
   2. Identifying the political message or narrative to be conveyed
   3. Finding and annotating hostile terms
   4. Finding claims that can be fact-checked (see Chapter 3 Fact-checking)
   5. Hostile narrative or not: Whether the content of the ad is hostile: The easiest way to determine whether a political ad contains a hostile narrative is to look for the presence of hostile language. In addition, or in the absence of hostile terms, an advertisement may be considered hostile if its context and audio-visual elements are likely to discredit or portray a political opponent as an enemy in a disproportionate manner. In controversial cases, it is necessary to develop an analytical consensus by examining compatibility with the defined concept. In such cases, decisions were made on the basis of independent classifications by two or three analysts.

After categorizing each ad, we created the report using the following criteria and content elements.

1. Summarizing the total amount of money spent on ads promoting a hostile narrative, based on the theme of the ads. Both Meta and Google give a spending range for individual ads. So we don't know exactly how much they spend. For aggregation, we have used the lower limit, so our results are a lower estimate of the actual spend. Neither the upper limit nor the statistical mean of the range could be used because there is no upper limit in the top category. For example, in the case of Meta, the top category includes ads with a minimum spend of 1 million HUF.
2. Prioritization of topics based on the amount of money spent.
3. Aggregating, contextualizing, describing and analyzing the narratives associated with each theme in its political and social context. Describing deviations from previous trends and narratives.
4. In the case of fact-checkable claims, manipulative claims were compared with Lakmusz’s fact-checking articles.
5. Listing and selection of hostile terms.
6. Hostile narratives that were disseminated in small quantities were briefly presented in a separate section entitled ‘other narratives’ below the top 4-5 themes. Borderline cases where it was not possible to reach an analytical consensus, but the volume of advertising associated with the content and the amount spent were also mentioned in the analysis.
7. For the analysis, we created interactive infographics showing the main hostile narratives of the given period, the advertisers and a selection of relevant ads.

Suggested readings:

Political Capital: Fidesz & Co. flood social media with anti-Western hostile disinformation in Hungary’s election campaign, reaching EU spending records (link)

Political Capital: Elsöprő fölénnyel zárta a közösségimédia-hirdetések versenyét a kormányoldal (link)

2. Quantitative analysis of ads - who is spending and how much on political ads on Meta and Google

Retrieval of aggregated spending data by advertiser

As explained above, information on the amount spent on a specific, individual ad is only available for its entire lifecycle and is not reported accurately by the platforms, but only for a range of amounts. As a result, this data is not suitable for building a picture of the amounts spent by advertisers in a given period. Fortunately, both Meta and Google produce aggregates that can be used for this purpose.

• Google produces weekly reports summarizing political advertiser spending in each country for a fixed seven-day period from Sunday to Saturday. These can be downloaded in .csv format from the Google Ads Transparency Center. Google updates the database every Tuesday with new data for the period from the previous Sunday to Saturday. The downloadable compressed file contains all previous weekly totals.
• In addition to the weekly reports, Meta also provides 1, 30 and 90 day aggregates. However, these reports are only available for one week in the Ad Library, unlike Google, which adds fresh weekly summaries to its downloadable database and makes historical data available for all weeks. Meta also has a three-day delay in producing summary reports.

In order to combine Meta’s and Google’s advertising data, we obtained weekly aggregates from Meta, always on a Tuesday, for the seven days from the previous Sunday to Saturday. This allowed us to combine the amounts spent on ads placed by the same advertiser on Meta and Google.

Categorization of advertisers

After data collection, the first step in processing was to categorize the advertisers. In the case of Google, this was easy because there are far fewer political advertisers than on Meta and they are easily identifiable (at least in the Hungarian social media space). In the case of Meta, the categorization is much more difficult. There are two variables that determine the classification: the page on which the ad ran and the sponsor that paid for the ad. A sponsor typically runs ads on multiple pages, and there are pages where multiple sponsors pay for the ads. There are thousands of page-sponsor pairs, each of which is subject to a categorization procedure.

We have created four main categories:

1. Political parties. We classified advertisers on Google and page-funding pairs on Meta as belonging to a particular political party if the advertiser, page, or sponsor was the party itself or a politician, candidate, or affiliate of the party.
2. This is one of the largest pro-government proxy organizations in Hungary. Megafon trains, coordinates, finances, and promotes pro-government social media ’influencers’. For this reason, it has its own category. On Google, Megafon appeared as an independent advertiser, on Meta there were 15 pages with ads paid by Megafon.
3. Civil Union Forum (CÖF). This is a government-organized NGO (GONGO) to which Fidesz usually outsources its negative campaigning. So its role is similar to Megafon’s. On Meta, it only ran ads on its own page, while on Google the ads were paid for by the CÖF-owned “Kövess minket 2022 Kft.”.
4. Media ads were divided into three subgroups depending on whether they (a) belong to the government-controlled media conglomerate, (b) are associated with an opposition actor, or (c) can be considered independent media.

These are the groups and subgroups into which the advertisers were classified. Since new advertisers were constantly appearing in the weekly ad reports of the two platforms, the categorization process had to be carried out on a weekly basis throughout the project, not just once. We categorized advertisers who spent at least HUF 5,000 on advertising in a given week.

Advertisers that did not fit into the above categories were not analyzed further. Some of these advertisers were not political actors at all and their ads did not contain such messages. They were included in the directory of advertisers because of Meta's broad definition of political advertising. There were also advertisers who advertised political content, but it was not possible to reliably determine which party they belonged to. This would have required a lengthy investigation with no guaranteed results, so it was not within the scope of our current project to analyze such advertisers. Overall, we were able to classify 88% of political ads (based on spending) into one of the four categories during the period under review, while 12% were classified as other advertisers, which were not analyzed further.

Based on the categorization, we created various aggregations and presented them in an interactive infographic in Hungarian and English, illustrating the relative spending of the actors and the change in spending over time. The infographic was updated on a weekly basis immediately after the weekly spend reports were retrieved and processed.

The bi-weekly reports on qualitative results were accompanied by results and findings on aggregate spending. This gave the public a complete picture of how the amount of money spent on political advertising evolved over the course of the campaign, and how advertisers and narratives changed.

Suggested readings:

Political Capital: Fidesz & Co. flood social media with anti-Western hostile disinformation in Hungary’s election campaign, reaching EU spending records (link)

Political Capital: Elsöprő fölénnyel zárta a közösségimédia-hirdetések versenyét a kormányoldal (link)

 

3. Fact-checking

Lakmusz, the project's fact-checking partner, is a verified signatory of the IFCN and EFCSN, and fully adheres to the standards of these organizations (including a commitment to non-partisanship and fairness, transparency of sources, transparency of funding and organization, transparency of methodology, honest correction policy). Its detailed fact-checking methodology is publicly available on its website. The team investigates dubious claims that are viral, impactful and/or potentially harmful to the public. Fact-checks are based on multiple pieces of evidence and non-partisan, primary-source materials. Journalists are transparent about the steps taken throughout the debunking process (including links, embeds, screenshots) and disclose any potential conflicts of interest. The subjects of articles always have the right to comment.

Lakmusz followed the same fact-checking methodology throughout the project, with additional considerations regarding the selection of topics and the calculation of the total amount spent on promoting false, misleading information.

Choice of topics

• Priority is given to debunking false claims over confirming truthful content.
• Despite the disparity in resources between actors connected to the governing party when it comes to promoted content, the newsroom remains fully impartial and will look at claims from across the political spectrum, applying the same standards regardless of who has made the claim.
• The project's research partners maintain an internal ad transparency database that is updated weekly. Researchers regularly flag ads with hostile narratives and suspicious claims that they deem worthy of more detailed fact-checking. Lakmusz takes the expert advice of the researchers into account when selecting claims to review. The Lakmusz newsroom and its members have the final say in the selection of advertised content to be fact-checked, and they have full editorial independence over the content of the fact-checks.

Calculation of the total amount spent on advertising false, misleading information:

• A political ad may contain more than one false/misleading claim, and a fact-checked claim may appear in more than one political ad. We publish one fact-check per claim and keep track of the claims we check in our internal ad transparency database.
• When we publish a fact-check, we collect all political ads containing the fact-checked claim (up to the day the fact-check is published), using our internal ad transparency database and keyword searches of VLOPs' ad libraries.
• To calculate the total amount spent on false/misleading information in a given time period, Lakmusz revisits all political ads flagged as "suspicious" by researchers in the internal ad transparency database and records which fact-checked claim (or claims) each ad contains. In this way, it is possible to calculate 1.) the amount of money spent on the ad containing each fact-checked claim, 2.) the amount of money spent on political ads containing one or more fact-checked claims.
• Since VLOPs' ad libraries currently only contain a range of amounts spent on a specific ad, we use the lower number for the calculation to assess the minimum spend.

Suggested readings:

Lakmusz: Megafon advertises for more than €100k that "pro-war left wing politicians" are leading us into the third world war (link)

Lakmusz: How does anyone who is not a friend of fidesz become a "leftist"? (link)

 

4. Measuring the impact of advertising using qualitative and quantitative methods

Focus group research

The focus group research explored participants' views on the role of the media, propaganda, and Megafon videos. The research consisted of four online focus group discussions with six participants per group. The interviews took place on 7 and 8 May 2024.

The participants were a mix of men and women, Budapest and rural residents.

Two groups of Fidesz voters and two groups of opposition voters were organized. There were two separate focus groups for opposition voters:

• DK-MSZP-P party alliance and Momentum voters (those parties that cooperated in the 2022 general elections)
• Tisza Party and MKKP voters (anti-establishment parties that positioned themselves outside the pro-government or opposition party cooperation)

The sessions lasted two hours.

Suggested reading:

Mérték: A magyarországi média és propaganda - Kvalitatív vizsgálat kormánypárti és ellenzéki szavazók körében (link)

Survey research

The goal of the survey was to find out how well respondents knew Megafon's ‘influencers’, what they thought of their videos, and the level of awareness among those who encounter paid content on social media.

The data was collected through a face-to-face survey of 1000 respondents. The sample is representative of the Hungarian adult population in terms of gender, age and education. The survey was conducted between 13 and 24 May.

The statistical margin of error is +/-3.2 percentage points for all respondents, and higher for smaller subgroups.

Suggested reading:

Mérték: Online tartalomfogyasztás és a Megafon - Kvantitatív kutatás eredményeinek összefoglalója (link)

5. Qualitative analysis of Megafon videos

To analyze Megafon's videos, we used the concept of the hostile narrative, which is also the basis of Political Capital's biweekly analyses. We conducted an in-depth analysis of Megafon's 105 most promoted videos to better understand this phenomenon and how these narratives are constructed in the videos.

Due to the size of the sample of 100 videos and the subject of our study, we chose a qualitative methodology. The framework for the analysis of the videos was developed after a pilot study over a period of two weeks. The videos were analyzed from a narrative, linguistic and audiovisual point of view.

Each video was first categorized by topic - during the period under study, these videos were usually produced in the context of a current event, so that each week focused on a different topic. Even when compiling the sample of videos, it was clear that the most expensive videos promoted during a given period were produced by different influencers, but on the same topic.

The narratives of the videos were differentiated on three levels. ''Actuality'' is the event or new information that forms the basis for the making of the video and in relation to which the speaker can revisit and reinforce the ''main narrative'', a recurring theme in public discourse. In the context of Megafon, these completely overlap with the main narratives and recurring agendas of the government. We treated as ''sub-narratives'' those texts that serve to link the main narrative and the current event. They usually highlight a particular aspect of the main narrative, with the aim of explaining the current event within a familiar framework.

In the videos, we have tried to identify the actual (currently targeted) and the arch-enemies (who embody the well-established enemy concepts of the main narratives), and to examine how they are constructed by the creators of the videos, what linguistic and visual tools they use to depict them. We also looked to see whether the " bad ones " are contrasted with the "good ones", the "enemies" with the “allies”, and if so, how these characters are constructed.

A crucial aspect was also the analysis of the disinformation tools used in the videos. What fears and stereotypes are used to construct the concept of the enemy? What is the threat they portray and what is the role of the main and actual enemies in the video? We also tried to find out and monitor whether they were using the tool of distorting the facts, or whether there was fundamentally false information or fake news in the video. To do this, we relied on Lakmusz's fact-checking.

We tried to explore the purpose of the videos by analyzing their conclusions. We looked at whether they point to an actor who could be the solution to the problem and how this relates to the "good guys/allies" in the narratives of the videos. Does the narrator articulate a lesson, suggest a solution, or call the viewer to a specific action? As a special aspect, we examined whether the video refers to or mobilizes for the European Parliament or local elections.

Megafon videos are a new genre, so we also looked at the linguistic and visual world of these videos. We noted the style, the way the video/influencer speaks, the setting, the framing, the background, and the situation in which the influencer appears. We collected common terms, phrases, or slogans that are already well established in the public sphere. We paid particular attention to extreme or polarizing expressions, but also included popular motifs, idioms, and slang because of their frequency. We also looked for the presence and use of various linguistic tools, such as metaphors or rhetorical questions. Similarly, we examined the audiovisual context of the videos, the use of different visual techniques, musical accompaniment and sound effects.

Suggested reading:

Mérték: “A baloldal új messiása” – Magyar Péter kampány visszakapcsolva (link)

 

 Disclaimer

A consortium led by Political Capital and including Lakmusz and Mérték Médiaelemző Műhely has won a €143,000 grant from the European Media and Information Fund (EMIF) for the implementation of the project on electoral disinformation. Any content supported by the EMIF is the sole responsibility of the author(s) and does not necessarily reflect the views of the EMIF or of the Fund's partners, the Calouste Gulbenkian Foundation and the European University Institute.