GLiClass: One Pass, Many Labels

Classification is the first decision

A classifier often sits before retrieval, routing, moderation, or generation. If it is slow, every downstream request waits. If it is brittle, its early mistake propagates.

GLiClass is designed for the awkward middle ground: richer than a similarity score, lighter than asking a generative LLM to follow a classification prompt, and flexible enough to accept labels at inference time.

Many labels should not mean many full model passes. The model should see the text and the label set together, then score the candidates in one forward pass.

This article follows the model from input construction to label scoring, then compares its scaling and few-shot behavior with the paper’s baselines. The visualizations use small deterministic examples unless they explicitly show reported benchmark values.

One encoder, many interacting labels

GLiClass adapts the GLiNER idea to sequence classification. The primary design is a uni-encoder: prepend a special LABEL marker to each class label, concatenate the labels with the input sequence, and run one bidirectional transformer encoder.

The model then separates representations into a pooled text vector $t$ and pooled label vectors $c_k$ . A dot product or small learned scorer produces a logit for each label:

s_{b,k}=\frac{t_b^\top c_{b,k}}{\tau} \qquad\text{or}\qquad s_{b,k}=g([t_b;c_{b,k}]).

Construct the label-conditioned sequence

Candidate labels4

one bidirectional pass

The special LABEL marker anchors each candidate. Label tokens share attention with the text and with one another.

Change the label count and inspect the single sequence that carries text, labels, and inter-label context. The score chips are illustrative.

Joint processing lets labels communicate. A “refund” label can be interpreted differently when “billing dispute” or “account closure” is also present. Pairwise cross-encoders instead repeat text-label processing, while embedding methods usually compare independently produced vectors.

Compare the architectural choices

→

Toggle the encoder arrangement to see what is shared, what interacts, and how label count changes the work.

Why not just use a generative LLM?

Generative models are flexible, but the paper identifies inconsistent instruction following and higher computational cost for classification. GLiClass keeps labels as explicit inputs and produces scores directly, avoiding autoregressive label generation.

Train the boundary, not a fixed taxonomy

A zero-shot classifier must learn how text and candidate labels relate, not memorize one immutable output head. GLiClass uses a staged recipe: a 1.2M-example pre-training corpus, reinforcement-learning fine-tuning with a PPO-style objective, and post-training streams focused on logic/NLI and word-count patterns.

The total RL objective combines policy, value, KL, and entropy terms:

\mathcal{L}_{total}=\mathcal{L}_{PPO}+\mathcal{L}_{value}+\mathcal{L}_{KL}+\mathcal{L}_{entropy}.

The paper also applies layer-wise attention re-weighting and token-level contrastive learning. The logic/NLI stream is especially important: formal entailment and contradiction examples sharpen the distinctions that semantic similarity alone can blur.

Few examples move the decision boundary

Examples per label0-shot

Average F1 from the paper’s Table 5. The path between reported endpoints is illustrative.

At 8 examples, the paper reports the largest relative gains for edge (+50.0%) and modern-base (+47.1%).

Drag the examples-per-label control. The curves mirror the paper's reported average F1 values for zero-shot and eight-shot settings; intermediate points are illustrative.

With eight examples per label, the paper reports large average relative gains for smaller variants: about 50% for edge and 47.1% for modern-base. Larger models improve too, but by smaller percentages. This is a practical trade: few-shot adaptation can buy a new domain without retraining a large generative model.

The reported few-shot gains are benchmark averages. They do not guarantee calibration on a new label set, and they do not remove the need to evaluate threshold choice and class imbalance for a real application.

More labels, less repetition

The paper evaluates inference on a single NVIDIA A6000 with batch size 1, sweeping label counts from 1 to 128 and input lengths of 64, 256, and 512 tokens.

GLiClass throughput declines mildly as the label set grows because labels share one forward pass. For gliclass-edge-v3.0, throughput moves from 103.81 to 82.64 examples/s between 1 and 128 labels. For gliclass-large-v3.0, it moves from 19.05 to 17.60 examples/s.

See the scaling shape

Model trace

Reported examples/s on one A6000 GPU, batch size 1. Hover bars for values.

1108050200

GLiClassdeberta-v3-base cross-encoder

Reported A6000 throughput at one and 128 labels. Cross-encoders degrade much more sharply because they process text-label pairs sequentially.

At the same endpoints, deberta-v3-base-zeroshot-v2.0 drops from 24.55 to 0.47 examples/s, approximately 52× slower at 128 labels than at one. The exact slope depends on sequence length and implementation, but the architectural contrast is the point.

GLiClass still consumes context for the label set. The paper notes that very large sets can hit a context limit around 1024 tokens, requiring truncation or batching, and that performance can vary by dataset.

Accuracy and speed are a frontier

Across the paper’s benchmark table, the largest GLiClass model reports the highest average F1: 0.7193. Base reaches 0.6764, modern-large 0.6197, modern-base 0.5577, and edge 0.4900. Throughput runs in the opposite direction: edge averages 97.29 examples/s on the A6000, while large averages 25.22.

Choose a point on the model frontier

Quality preferencebalanced

Every dot is a paper-reported model overview value.

throughput →F1 ↑

Policy: minimize weighted distance to a quality/speed target. This recommendation is an explanatory overlay, not a benchmark result.

Reported zero-shot average F1 and A6000 throughput from Table 2. Drag the quality preference to select a model; the recommendation is a deterministic policy over the paper's values.

Compared with cross-encoders, the paper reports GLiClass-large at 0.7193 average F1 versus 0.6821 for the strongest listed cross-encoder, a +0.037 absolute difference. The comparison is benchmark-specific and depends on the selected model variants and settings.

Dataset behavior is not uniform

Reported zero-shot F1. The selected variant is not guaranteed to win every dataset: compare the bars, then inspect the average separately.

Reported zero-shot F1 for selected datasets. Different variants win different tasks, so the average is a summary rather than a universal ordering.

Results vary by dataset. For example, base leads on ag_news while modern-large is close on sst5 and imdb. A deployment should measure its own labels, thresholds, and error costs.

The reusable classifier shape

GLiClass makes a useful bargain:

labels remain natural-language inputs, so the taxonomy can change;
text and labels interact in one encoder pass;
independent scores support multi-label decisions;
smaller variants can adapt with a handful of examples;
inference cost grows more gently with label count than pairwise cross-encoders.

The boundary is equally useful. More labels still consume context, zero-shot scores still need calibration, and accuracy still depends on the dataset. GLiClass is not a claim that one classifier wins every task. It is a generalist sequence-classification interface designed to make flexible labels practical at production latency.

Read the source

Ihor Stepanov, Mykhailo Shtopko, Dmytro Vodianytskyi, Oleksandr Lukashov, Alexander Yavorskyi, and Mykyta Yaroshenko. “GLiClass: Generalist Lightweight Model for Sequence Classification Tasks.” arXiv:2508.07662, 2025. Paper · Python library · Model collection