Political Science LLM Benchmark

Hanno Hilbig, University of California, Davis
Results as of November 2026

On this page, I compare the performance and cost of 25 language models – 13 open-weight models as well as 12 commercial models that are accessed through an API – on text classification tasks from political science. I rely on 46 coding tasks drawn from published papers, replication archives and public data sets, covering relevance, stance, tone, events, claims and topics. For each task, every model codes the same 100 texts (50 for the six tasks with the longest texts), and I compare the labels assigned by the model to the labels assigned by human coders in the original data.

Overall, I find that differences between models are small. Gemini 4 Argon has the highest mean F1 across tasks (0.703), and the best open-weight model, Gemma 4 31B (W4A16), reaches 0.694, which is higher than seven of the 12 API models. In addition, differences of a few hundredths in the mean F1 often reverse on individual tasks – researchers who want to choose a model should therefore look at the tasks that are closest to their own application.

Overall performance

Overall performance. Each row shows one model. Points mark the mean F1 across the 46 tasks, grey bars mark 95% intervals that I obtain from resampling tasks. Open-weight models are blue, API models are grey.
Each row shows one model. Points mark the mean F1 across the 46 tasks, grey bars mark 95% intervals that I obtain from resampling tasks. Open-weight models are blue, API models are grey.

Cost and performance

Cost and performance. Higher prices buy relatively small gains in accuracy. Gemini 4 Argon costs about 65 times as much per text as DeepSeek V4.1 Flash, the cheapest API model, and scores 0.042 higher (at batch prices, the ratio is about 32 instead of 65). Five of the 12 API models score above the best open-weight model that runs on one GPU (dashed line), for which there is no per-text charge. Costs are per 1,000 texts at standard prices at the time of each run, and hollow points are estimates based on token counts rather than provider bills.
Higher prices buy relatively small gains in accuracy. Gemini 4 Argon costs about 65 times as much per text as DeepSeek V4.1 Flash, the cheapest API model, and scores 0.042 higher (at batch prices, the ratio is about 32 instead of 65). Five of the 12 API models score above the best open-weight model that runs on one GPU (dashed line), for which there is no per-text charge. Costs are per 1,000 texts at standard prices at the time of each run, and hollow points are estimates based on token counts rather than provider bills.

API and open models by task

API and open models by task. Each row shows one task, and points show the best API score minus the best score among the 12 open-weight models that run on one GPU. An open model matches or beats the best API model on 14 of the 46 tasks (blue points), although this number is somewhat inflated in favor of open models, since on every task I pick the best out of 12 open models. If I compare a fixed pair of models instead – Gemma 4 31B and Gemini 4 Argon – Gemma matches or beats Gemini on 18 tasks, and Gemini leads by 0.009 F1 on average.
Each row shows one task, and points show the best API score minus the best score among the 12 open-weight models that run on one GPU. An open model matches or beats the best API model on 14 of the 46 tasks (blue points), although this number is somewhat inflated in favor of open models, since on every task I pick the best out of 12 open models. If I compare a fixed pair of models instead – Gemma 4 31B and Gemini 4 Argon – Gemma matches or beats Gemini on 18 tasks, and Gemini leads by 0.009 F1 on average.

Performance by annotation type

Performance by annotation type. Mean F1 within each of five annotation types, with each task weighted equally. Models appear in the same order in every panel, ranked by their overall mean.
Mean F1 within each of five annotation types, with each task weighted equally. Models appear in the same order in every panel, ranked by their overall mean.

Coding complexity

Coding complexity. The gap between API and open models increases with coding complexity. If I compare the best open model on each task with the best API model on that task, the open model is behind by 0.02 F1 on the 17 low-complexity tasks, while it trails by about 0.04 F1 on medium- and high-complexity tasks. High-complexity tasks allow several labels per text or use at least eight labels in practice, and medium-complexity tasks use at least three labels or have a prompt of 300 words or more. To count the number of labels in practice, I use the exponential of the entropy of the gold labels, which gives less weight to rare labels than to common ones.
The gap between API and open models increases with coding complexity. If I compare the best open model on each task with the best API model on that task, the open model is behind by 0.02 F1 on the 17 low-complexity tasks, while it trails by about 0.04 F1 on medium- and high-complexity tasks. High-complexity tasks allow several labels per text or use at least eight labels in practice, and medium-complexity tasks use at least three labels or have a prompt of 300 words or more. To count the number of labels in practice, I use the exponential of the entropy of the gold labels, which gives less weight to rare labels than to common ones.
All 25 models and open-model speed

The first figure repeats the overview for all 25 models. The second figure compares the speed of the 12 open-weight models for which I recorded generation time.

Overall performance, all models

Overall performance, all models. Each row shows one model. Points mark the mean F1 across the 46 tasks, grey bars mark 95% intervals that I obtain from resampling tasks. Open-weight models are blue, API models are grey.
Each row shows one model. Points mark the mean F1 across the 46 tasks, grey bars mark 95% intervals that I obtain from resampling tasks. Open-weight models are blue, API models are grey.

Quality and generation time

Quality and generation time. Each point shows one open-weight model, with mean F1 plotted against generation time per text on one RTX PRO 6000 Blackwell GPU. The models in each panel coded the same texts with identical settings, but in the first panel vLLM processed as many texts at once as memory allowed, while in the second panel I limited it to 16 texts at a time, so I would only compare times within a panel and not across the two panels. In the first panel, the 9 models need between 0.3 and 3.0 minutes per 1,000 texts, and in the second panel the 3 models need between 0.6 and 3.2 minutes (not counting the time it takes to load a model or to wait in the queue).
Each point shows one open-weight model, with mean F1 plotted against generation time per text on one RTX PRO 6000 Blackwell GPU. The models in each panel coded the same texts with identical settings, but in the first panel vLLM processed as many texts at once as memory allowed, while in the second panel I limited it to 16 texts at a time, so I would only compare times within a panel and not across the two panels. In the first panel, the 9 models need between 0.3 and 3.0 minutes per 1,000 texts, and in the second panel the 3 models need between 0.6 and 3.2 minutes (not counting the time it takes to load a model or to wait in the queue).
How much the ranking depends on single tasks

Rank when one task is left out

Rank when one task is left out. Points show the rank of each model by mean F1, and grey bars show the range of ranks that a model reaches if I leave out one of the 46 tasks at a time and compute the mean over the remaining 45. Gemini 4 Argon is ranked first in all 46 cases, while the rank of Qwen3.6 27B (FP8) changes by up to 4 places, which is the largest change for any of the models.
Points show the rank of each model by mean F1, and grey bars show the range of ranks that a model reaches if I leave out one of the 46 tasks at a time and compute the mean over the remaining 45. Gemini 4 Argon is ranked first in all 46 cases, while the rank of Qwen3.6 27B (FP8) changes by up to 4 places, which is the largest change for any of the models.
The 46 tasks

Each line gives the task, the study or dataset it comes from, and what the model codes.

  • arslan claimbuster checkworthy (Arslan et al. 2020 (ClaimBuster)): claim type, one of 3 categories.
  • ballard incivility (Ballard 2022): uncivil, yes or no.
  • benoit manifesto economic position (Benoit et al. 2016): economic position, one of 5 categories.
  • benoit manifesto social position (Benoit et al. 2016): social position, one of 5 categories.
  • benson tucker unsc government stance (Benson and Tucker (UNCR) and CR-UNSC 2024): government stance, one of 4 categories.
  • bestvater kavanaugh stance (Bestvater and Monroe 2023): pro kavanaugh, yes or no.
  • bestvater wm stance (Bestvater and Monroe 2023): pro womens march, yes or no.
  • bevan uk pmqs policy topic (Bevan and John 2016 (UK Policy Agendas Project, Prime Minister's Questions)): policy topic, one of 21 categories.
  • brandt gtd attack type (Brandt et al. 2025): attack type, one of 8 categories.
  • burnham covid threat minimization (Burnham 2025): threat minimizing, yes or no.
  • burnham trump stance (Burnham 2025): stance toward trump, one of 3 categories.
  • cap crs policy topic (Policy Agendas Project 2025 and Congressional Research Service products): policy topic, one of 21 categories.
  • cap executive order policy topic (Policy Agendas Project 2025 and Federal Register executive orders): policy topic, one of 21 categories.
  • cap party platform policy topic (Policy Agendas Project 2025): policy topic, one of 21 categories.
  • chalkidis trump populism (Chalkidis, Brandl and Aslanidis 2026): populism, one of 4 categories.
  • dicocco manifesto populism (Di Cocco and Monechi 2022): populist, yes or no.
  • douglass icbe sentence event type (Douglass et al. 2024): event type, one of 5 categories.
  • erhard popbert populism (Erhard et al. 2025 (PopBERT)): 4 yes-or-no labels (anti elitism, people centrism, left wing, ...).
  • erlich ati topics (Erlich et al. 2022): 6 yes-or-no labels (Activities, Budget, Evaluation, ...).
  • gilardi stance (Gilardi et al. 2023): stance, one of 3 categories.
  • gpd speech populism (Hawkins et al. 2025 (Global Populism Database v2.1)): populism, one of 3 categories.
  • halterman keith bfrs (Halterman and Keith 2026): event type, one of 12 categories.
  • halterman keith cmp (Halterman and Keith 2026 plus Manifesto Project 2025): policy domain, one of 7 categories.
  • heseltine politician tweet tone (Heseltine and Clemm von Hohenberg 2024): tone, one of 3 categories.
  • horak czech propaganda (Horák et al. 2024 (MU Propaganda dataset)): 5 yes-or-no labels (blaming, opinion, labeling, ...).
  • kiesel hyperpartisan article (Kiesel et al. 2019): hyperpartisan, yes or no.
  • lemens gallego tweet position3 (Le Mens and Gallego 2025): position, one of 3 categories.
  • liu gvfc frame (Liu et al. 2019 (Gun Violence Frame Corpus, CoNLL 2019)): frame, one of 9 categories.
  • mellon bes mii 2024 (Mellon et al. 2024): issue, one of 49 categories.
  • mets estonian immigration stance (Mets et al. 2024 (Estonian immigration stance)): stance, one of 3 categories.
  • muller fujimura campaign policy area (Müller and Fujimura 2025): policy area, one of 12 categories.
  • muller proksch polnos nostalgia (Müller and Proksch 2024 (PolNos, BJPolS)): nostalgia, yes or no.
  • ornstein scotus sentiment (Ornstein et al. 2025): sentiment, one of 3 categories.
  • parlasent en sentiment (Mochtak, Rupnik and Ljubešić 2023 (ParlaSent 1.0, English)): sentiment, one of 3 categories.
  • plover cameo event (Halterman et al. 2023): event type, one of 18 categories.
  • politicause causal relation (Garcia Corral et al. 2024): causal relation, yes or no.
  • rheault line of fire incivility (Rheault, Rayment, and Musulan 2019): uncivil, yes or no.
  • rivlin angert pdd delegitimization (Rivlin-Angert and Mor-Lan 2025 (PDD, EMNLP 2025)): delegitimization, yes or no.
  • scdb opinion issue area (Spaeth et al. (Supreme Court Database), via Chalkidis et al. 2022 (LexGLUE)): issue area, one of 12 categories.
  • theocharis dynamics incivility (Theocharis et al. 2020): uncivil, yes or no.
  • thomas qevasion clarity (Thomas et al. 2024 (QEvasion / CLARITY)): clarity, one of 3 categories.
  • toxicity protests es (González-Bustamante 2024): toxic, yes or no.
  • trager mfrc political moral (Trager et al. 2022 (Moral Foundations Reddit Corpus)): 5 yes-or-no labels (Care, Equality, Proportionality, ...).
  • twitcivility impoliteness (Pendzel et al. 2023): impolite, yes or no.
  • wesleyan creative ads 2022 (Zhang et al. 2025): tone, one of 3 categories.
  • yan bernhard offensive (Yan and Bernhard 2024): offensive, yes or no.
Does the ranking depend on the kind of task?

I also split the tasks into groups that I defined before I looked at the results, i.e. by language and by the type of label, and for the open-weight models I split the texts of each task into a shorter and a longer half – since the groups differ in other ways as well, I only look at whether the order of the models changes.

  • English tasks (36), and shorter compared to longer texts: the order hardly changes.
  • Tasks that are not only in English (10, in eight languages): Gemma 4 31B (W4A16) ranks first and Claude Haiku 5.5 second, while GPT-6.1 Sol falls from third to 20th and Claude Opus 5.5 from second to 11th. Six of the 10 tasks are yes/no decisions, though, and I would not put too much weight on the exact ranks.
  • Yes/no decisions, three to five classes and seven or more classes: the models that reason before they answer are ahead mostly on the tasks with seven or more classes (by about 0.07 compared to the average model), and not on the yes/no decisions.

Researchers who code texts in other languages should therefore test a few models on their own data (details).

Not evaluated and excluded models

The models below are not ranked, either because I was not able to run them, in which case they are listed as not evaluated together with the reason, or because a run finished but violated the rules of the benchmark, in which case they are listed as excluded.

The open-weight models on this page are a selection rather than a complete list, and whether I can run a given model depends not only on its size but also on its weight files, its quantization, software support and GPU memory – if a model is missing, this does not say anything about its quality.

  • No additional candidate status has been recorded.

Results from the November 2026 release, in which I compared 43 models on 33 tasks (24 of which are also part of the 46 tasks that I use here), are still available as release refresh_20261106, although the two sets of results are not directly comparable, since the tasks, the texts and many of the models differ.

Where the models ran

Methods and limitations

Sample. For each task, I draw 100 texts (50 for the six tasks with the longest texts) at random, stratified by task, from the texts that the open-weight models coded, and I score every model on the same 4,298 texts. All models receive the same texts, gold labels and prompts. I only reuse earlier predictions if inputs and settings match exactly.

Scoring. For binary tasks, I use the F1 score for the positive class, for tasks with several binary labels I use the average of the per-label F1 scores, and for categorical tasks with a single label I use macro F1, which gives equal weight to each class. Classes that do not appear in the gold labels of a task are left out of its macro F1. Unusable answers are counted as incorrect, while infrastructure failures are recorded separately.

Limits. All results are based on 46 tasks. Since most tasks are taken from published data sets, some of the texts or labels may have been part of the training data of the models, which would increase their scores, and I am not able to rule this out. To get some sense of how large this problem is, I compared the 13 open-weight models on tasks whose source was published before the model was released and on tasks that were published afterwards, net of differences between models and between tasks, and I do not find that the models do better on the older tasks – the difference is about 0.00 in F1, with a 95% interval from −0.06 to 0.06 (details), although this check does not tell me whether the models have seen the texts themselves. Before using a model at scale, researchers should validate it on labeled examples from their own task.

How to cite

Hilbig, Hanno. 2026. Political Science LLM Benchmark. Results as of November 2026, release tasks46_20261119. https://www.hannohilbig.com/llm-benchmark/.