Understanding GLiFormer’s Benchmarks and Schema-Driven Extraction

Wait 5 sec.

Overviewgliformer-large-v1 is a 575.6-million-parameter, Apache-2.0 model from knowledgator for schema-driven information extraction and text representation. It uses a shared DeBERTa encoder with task heads for named entity recognition, classification, joint relation extraction, structured records, and text embeddings. Its layout-aware architecture supports text and document-layout inputs, including PDF-oriented processing, but the published examples and evaluation results focus on English text rather than visual document understanding. You run it through the gliformer Python library or the GLiFormer source framework in Python 3.10 or newer. The most important consideration is its breadth: one checkpoint accepts labels and schemas at inference time, but its quality varies by task and dataset. Reported mean strict entity F1 is 50.91 across 26 NER datasets, mean macro-F1 is 75.03 across 13 classification datasets, joint relation extraction is weak on several benchmarks, and the strong 91.10 structuring score uses order-free, boundary-tolerant JSON evaluation rather than exact JSON matching. No context window, maximum sequence length, training-step count, VRAM requirement, inference latency, or checkpoint-specific training dataset is provided.Best use casesSchema-driven entity extraction from English text. Use the model when entity types vary by project and you want to provide labels at inference time instead of training a separate fixed-label NER model. For example, extract person, organization, and location from "Alice works at Acme in London." and receive each entity’s text, label, character offsets, and score. The shared DeBERTa encoder and task-conditioned interface support custom entity labels, but benchmark results show substantial variation: strict F1 reaches 80.16 on WikiNeural and 71.48 on CrossNER politics, while it falls to 13.81 on HarveyNER and 21.99 on FabNER.Text classification with custom candidate classes. Use classify for jobs such as assigning support tickets to product or support, detecting sentiment, or routing documents among business categories. The model accepts candidate labels at inference time and supports named groups such as separate sentiment and topic label sets. It performs well on some conventional datasets, including 97.95 macro-F1 on Enron spam, 96.77 on Financial Phrasebank, 93.93 on IMDB, and 92.97 on SST-2, but performance is much lower on SST-5 at 40.33, Emotion at 48.07, and the SOTU dataset at 49.00.Structured record extraction from semi-structured prose. Use structure when the output must contain fields and parent-child relationships, such as converting company descriptions into nested companies, departments, and employees. The decoder can produce Python dictionaries from simple field lists or nested Pydantic schemas. The reported multilevel structuring score is 91.10 on 500 examples, with scores from 89.94 at JSON depth 3 to 95.11 at depth 4. Treat this as source-grounded extraction, not factual verification: Pydantic validation checks shape and types, not whether extracted values are correct.Joint relation extraction with a closed schema. Use inference with joint_relations when you need entities and relations in one configured task, such as extracting (Alice, works_at, Acme). This checkpoint has a joint relation head, so the schema must provide entity and relation labels together. It is not a strong choice for high-recall relation mining: reported micro-F1 is 12.78 on DocRED, 12.61 on CrossRE, 24.08 on FewRel, and 35.83 on zero-shot CoNLL04.Embedding short text for similarity workflows. embed_text returns 1,024-dimensional vectors that you can compare with cosine similarity. This fits semantic search prototypes, duplicate detection, clustering, and retrieval features where a single local encoder is useful alongside extraction tasks. The provided material does not include embedding benchmark scores, so validate retrieval quality on your own corpus before selecting it for production search.LimitationsThe model does not provide a guaranteed extraction of every fact. NER can omit mentions, select incorrect spans, or assign incorrect labels. Its strict NER score requires both span and type to match, and the dataset results range from 13.81 to 80.16 F1. Labels, schema wording, domain, input length, and thresholds affect predictions.Relation extraction is a major weakness in the published results. The reported runs use predicted entities, and the scores are not controlled comparisons between the base and large releases because they use different-sized subsets. The model also lacks an open relation head: the convenience method predict_relations requires one, so this checkpoint should use joint_relations instead.Structured extraction can attach records to the wrong parent, omit fields, or produce incorrect values. Its 91.10 score is not exact JSON match. Evaluation flattens JSON value paths, aligns records without requiring original order, and permits limited boundary repairs. Fixed record anchors and configured span width constrain extraction capacity.The layout-aware design supports text and document-layout inputs, but the supplied quality evidence does not establish performance on PDFs, other document layouts, or visual inputs. The checkpoint has no dedicated vision or audio capability. Results cover English tasks; quality for other languages is not established. The NER transfer groups also do not prove that every evaluated domain was absent from training.The model is large at 575.6M parameters. The documentation provides no VRAM minimum, latency, throughput, maximum context length, or practical batch-size guidance beyond the NER example using batch_size=8. CUDA attention kernels are optional; CPU inference uses eager attention. You should benchmark memory and latency on your target hardware.The Apache-2.0 license permits commercial use subject to the license terms. The provided material does not document additional dataset-specific restrictions, warranty terms, or safety evaluations. It also does not state whether the checkpoint is actively maintained.How it comparesgliner-x-largeChoose gliformer-large-v1 over gliner-x-large when you need one interface for NER, classification, joint relations, structured records, and embeddings, or when nested record extraction matters. Choose GLiNER X-Large when the primary requirement is flexible zero-shot NER and you want a model positioned as a practical alternative to fixed-label NER systems and larger language models. The supplied information does not provide matched parameter counts, latency, cost, or benchmark scores, so it does not support a numeric quality or speed ranking between them.modern-gliner-bi-large-v1.0Choose gliformer-large-v1 when multi-task extraction, classification, relation extraction, structured records, and embeddings must share one checkpoint. Choose modern-gliner-bi-large-v1.0 for NER workloads that benefit from its bi-encoder design: the model card describes a ModernBERT-large text encoder and BGE-base-en-v1.5 entity-label encoder, with precomputable entity embeddings and support for an unlimited number of entities at one time. The provided material gives no controlled speed, cost, or quality comparison.gliner_large-v2Choose gliformer-large-v1 when you need schema-driven structuring, classification, joint relation extraction, or embeddings in addition to NER. Choose gliner_large-v2 when the project is centered on flexible NER and you prefer the GLiNER model trained on the commercially permissive NuNER dataset. No matched benchmark, parameter, speed, or cost data is supplied, so claims of superiority would be unsupported.gliner2-large-v1Choose gliformer-large-v1 when joint relation extraction and 1,024-dimensional text embeddings are part of the same pipeline, or when you need its documented layout-aware architecture. Choose gliner2-large-v1 when its stated CPU-first design, schema-driven field types and constraints, and zero-external-dependency deployment fit your environment. Both target multi-task extraction, but the supplied information does not provide a controlled comparison of quality, latency, memory, or operating cost.gliner-multitask-large-v0.5Choose gliformer-large-v1 when nested Pydantic records, joint relations, document-layout support, or embeddings are required. Choose gliner-multitask-large-v0.5 when prompt-tunable multi-task extraction and NER are the main goals. Its description emphasizes a bidirectional transformer, generalization, compute efficiency, and zero-shot NER, but the supplied material does not establish matched quality, speed, or cost differences.The related research includes GLiNER2: Efficient Multi-task Information Extraction System, which is relevant when comparing schema-driven multi-task extraction designs.Technical specificationsModel: knowledgator/gliformer-large-v1Parameters: 575.6MArchitecture: Shared DeBERTa encoder with task heads; layout-aware design for text and document-layout inputsLibrary: gliformerFramework: GLiFormer Python frameworkPython: 3.10 or newerLicense: Apache-2.0Pipeline tag: token-classificationModel tag: Text-to-TextTasks: NER, text classification, joint relation extraction, structured extraction, text embeddings, and intended PDF/document-layout processingEmbedding output: 1,024-dimensional vectorsInstallation: pip install gliformer -U, or editable installation from the GLiFormer sourceOptional acceleration: CUDA attention kernels through pip install -e ".[flash]"CPU behavior: Eager attentionDevice setup: Move the model to CUDA when available; otherwise use CPUEvaluation scope: English text tasksReported aggregate results: NER: 26 datasets, 131,156 examples, mean strict entity F1 50.91CrossNER: 5 domains, 2,505 examples, mean strict entity F1 64.35Classification: 13 datasets, 79,828 examples, mean macro-F1 75.03Multilevel structuring: 500 examples, order-free boundary-tolerant JSON F1 91.10NER evaluation: Exact entity span and type must matchClassification evaluation: Macro-F1 averages class F1 within each dataset; micro-F1 equals accuracy in the single-label runsCoverage: 97.15% on 20 Newsgroups and 100% on the other listed classification datasetsStructuring evaluation: Flattened JSON value paths, record alignment without order requirements, and limited boundary repairs; not exact JSON matchRelation evaluation: Predicted entities; gold counts represent relation instances, not documentsMissing published specifications: No context window, maximum input length, VRAM requirement, inference latency, throughput, quantization option, model file format, training-step count, or checkpoint-specific training dataset is documented.Model inputs and outputsInputsPlain text strings for NER, classification, relation extraction, structuring, and embeddings.A list of text strings for batched extraction or embedding.NER label lists such as ["person", "organization", "location"].Classification label lists such as ["positive", "negative", "neutral"].Named classification groups such as {"sentiment": [...], "topic": [...]}.Joint relation schemas containing task names, entity labels, and relation labels.Structure schemas using field lists or nested Pydantic models.Optional threshold, with examples using 0.5.Optional NER batch_size; the example uses 8.Document-layout inputs are supported by the architecture, but the supplied examples do not define their input format or preprocessing pipeline.OutputsNER dictionaries containing text, label, start, end, and score.NER offsets use character positions, with an exclusive end.Classification label dictionaries containing class_name and score.inference returns a dictionary of task outputs, with one result per input text.Joint relation results expose head, relation, and tail values; the example reads from results["joint_relex"][0].Structured extraction returns Python dictionaries, including nested records for Pydantic schemas.validate_output=True applies Pydantic schema validation, but does not verify factual correctness.Embeddings are PyTorch tensors with shape [number of texts, 1024].Cosine similarity can be computed with torch.nn.functional.cosine_similarity.Getting startedInstall the package and run NER, classification, and structured extraction in one call:pip install gliformer -Uimport torchfrom gliformer import GLiFormerdevice = "cuda" if torch.cuda.is_available() else "cpu"model = GLiFormer.from_pretrained( "knowledgator/gliformer-large-v1", load_tokenizer=True,).to(device).eval()text = "Alice joined Acme as a software engineer in London."results = model.inference( text, entities=["person", "organization", "location"], classes=["business", "sports", "technology"], structures={"employee": ["name", "company", "role"]}, threshold=0.5,)print(results["ner"][0])print(results["classification"][0])print(results["structuring"][0])For NER with character offsets:entities = model.predict_entities( "Alice works at Acme in London.", ["person", "organization", "location"], threshold=0.5,)for entity in entities: print( entity["text"], entity["label"], entity["start"], entity["end"], entity["score"], )For joint relation extraction, use joint_relations rather than predict_relations:results = model.inference( "Alice works at Acme.", joint_relations={ "employment": { "entities": ["person", "organization"], "relations": ["works_at"], } }, threshold=0.5,)for relation in results["joint_relex"][0]: print( relation["head"]["text"], relation["relation"], relation["tail"]["text"], )Frequently asked questionsQ: Can I use gliformer-large-v1 commercially?A: The checkpoint uses the Apache-2.0 license, which permits commercial use subject to that license’s terms. The supplied documentation does not list additional dataset-specific restrictions.Q: What hardware or VRAM do I need?A: The model has 575.6M parameters, but no minimum VRAM or system-memory requirement is documented. It can run on CPU with eager attention, or on CUDA; optional CUDA attention kernels are available through the flash installation extra.Q: How fast is inference?A: The provided material gives no latency, throughput, or tokens-per-second measurements. The NER example supports a list of texts with batch_size=8, but that is an API example rather than a performance guarantee.Q: What input format does the model expect?A: Core examples pass plain Python strings plus labels or schemas. NER takes entity labels, classification takes candidate classes, joint relations take entity and relation labels, structuring takes field lists or Pydantic schemas, and embeddings take a list of strings.Q: Can I use it for PDF understanding?A: Its layout-aware architecture supports text and document-layout inputs and is intended to cover PDF layout processing. The supplied evaluations focus on English text, and no PDF preprocessing format or layout benchmark result is provided.Q: What are the main failure modes?A: The model can omit information, select incorrect spans, assign wrong labels, attach records to the wrong parent, and produce structurally valid but factually incorrect Pydantic outputs. Thresholds, schema wording, domain, labels, and input length affect results.Q: Is joint relation extraction supported?A: Yes. Use model.inference with joint_relations, supplying entity and relation labels together. The separate predict_relations method requires an open relation head and is not the correct interface for this checkpoint.Q: Can I fine-tune it?A: The model is distributed through the GLiFormer framework, which supports the checkpoint’s task heads and evaluation entry points. The supplied material does not document a fine-tuning command, training recipe, compute requirement, or checkpoint-specific training provenance, so you should inspect the framework before planning fine-tuning.Q: Is the model multilingual?A: The published quality results cover English tasks. The documentation does not establish quality for other languages.Q: Is it still actively maintained?A: The supplied information does not state the maintenance status. It identifies the maintainer as knowledgator and provides the model’s framework and evaluation entry points, but no release cadence or support commitment is documented.