Ouroboros detector v7: merged backbone, heads, calibrator, model card

.gitattributes

@@ -33,3 +33,8 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/architecture.png filter=lfs diff=lfs merge=lfs -text
+assets/benchmarks.png filter=lfs diff=lfs merge=lfs -text
+assets/generators.png filter=lfs diff=lfs merge=lfs -text
+assets/interpretability.png filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,114 @@
+---
+license: apache-2.0
+language:
+  - en
+  - fr
+library_name: transformers
+base_model: Qwen/Qwen3-1.7B-Base
+pipeline_tag: text-classification
+tags:
+  - ai-text-detection
+  - ai-generated-text
+  - token-classification
+  - authorship
+  - lora
+  - qwen3
+---
+
+<div align="center">
+<img src="assets/banner.png" alt="Ouroboros" width="100%">
+</div>
+
+# Ouroboros detector (v7)
+
+An open, **token-level detector of AI-written text**. For a document it returns character-aligned segments labelled `human`, `ai-assisted` or `ai-generated` with confidences, a weighted AI fraction, a `human` / `mixed` / `ai` verdict, and a 4-way "was this humanized?" probability (`human`, `ai-generated`, `humanized-ai`, `mixed-authorship`).
+
+> **Unofficial.** This is an independent, open reimplementation of the method of the *Pangram 4* technical report ([arXiv:2607.27183](https://arxiv.org/abs/2607.27183)). It is not affiliated with or endorsed by Pangram, and it is not their model, weights or data.
+
+> **Built end to end by Claude Opus 5.5.** The architecture implementation, data pipeline, training, evaluation, interpretability study, bias audit and this model card were designed, written and run end to end by Claude Opus 5.5 (Anthropic) as an autonomous coding agent, at the direction of the author, who set the goals and reviewed the results.
+
+- **Code, training pipeline, dataset builder, demo:** https://github.com/Jourdelune/ouroboros-detector
+- **Backbone:** [Qwen3-1.7B-Base](https://huggingface.co/Qwen/Qwen3-1.7B-Base) with a LoRA adapter (r = 32) **merged into the weights** (bf16, 3.4 GB), plus four single-layer heads.
+- **Languages:** evaluated on English and French; other languages appear in the training data but are not benchmarked.
+
+## Use
+
+```bash
+pip install "git+https://github.com/Jourdelune/ouroboros-detector.git"
+```
+
+```python
+from ouroboros.infer.predict import Predictor
+
+detector = Predictor.from_pretrained("Jour/ouroboros-detector")
+result = detector.predict(text)                  # texts of at least ~50 words
+print(result.document_label, result.weighted_ai_fraction)
+for seg in result.segments:
+    print(seg.start, seg.end, seg.label_name, seg.confidence)
+print(detector.humanizer(text))
+```
+
+The model needs the `ouroboros` package: it implements Repeat2 inputs, the four heads, sliding windows, calibration and CRF decoding. `AutoModel.from_pretrained` alone loads only the backbone.
+
+## Architecture
+
+One causal backbone fed the window twice (**Repeat2**, so every supervised token has seen the whole window) and four heads: segment (`D→15`, weighted AI fraction), tokenwise provenance (`D→3`), mixed-authorship (`D→2`), humanizer probe (`D→4`, stop-gradient). Windows of 512 tokens with stride 256, then calibration and a linear-chain CRF.
+
+<img src="assets/architecture.png" alt="Architecture" width="100%">
+
+## Evaluation
+
+Held-out test split of the project's corpus: 12,000 documents (4,531 human, 6,821 AI, 648 mixed), split by source document. Operating point chosen on a separate calibration split for a 0.5 % false-positive target.
+
+| metric | v7 |
+|---|---:|
+| False-positive rate | 1.35 % |
+| False-negative rate (AI or mixed missed) | 0.94 % |
+| AUROC | 0.9915 |
+| TPR @ 1 % FPR | 98.9 % |
+| Token accuracy | 98.4 % |
+| Token F1 — human / AI-assisted / AI-generated | 0.986 / 0.714 / 0.988 |
+| Mixed-document accuracy | 77.2 % |
+| 4-way humanizer head accuracy / recall on `humanized-ai` | 95.0 % / 56.8 % |
+
+<img src="assets/benchmarks.png" alt="Benchmarks" width="100%">
+
+These numbers are **in-distribution** (same corpus families as training) and **not comparable** to the Pangram report's, which uses a different, proprietary corpus and backbone.
+
+## Known limitations and biases
+
+<img src="assets/biases.png" alt="Known biases" width="75%">
+
+- **Higher false-positive rate** on French human text (7.0 % vs 1.0 % in English), on very short text (< 80 words: 6.5 %), and on human text formatted like an assistant answer (bullets / bold / headers: 8.8 %).
+- **Format and encoding shortcuts:** human text split into paragraphs is flagged 14.5 % of the time; inserting four Cyrillic look-alike letters into a human text makes it read as AI in 86 % of cases. Normalise confusable characters before inference.
+- **Not robust to adaptive adversaries:** a rewriting loop with access to the detector's score made 70–88 % of a small held-out set (50 French + 50 English passages) read as human while passing automatic fidelity checks.
+- Not supported: texts under ~50 words. `ai-assisted` is the weakest class (F1 0.71).
+- **Do not use a verdict as the sole basis for a decision about a person.** AI-text detectors can wrongly flag non-native writers and formal prose.
+
+## What it looks at (interpretability)
+
+Measured on held-out passages with 95 % intervals: a linear probe on the *untrained* Qwen3-1.7B-Base already separates classes at 95 %; fine-tuning raises it to 98.5 % and makes it language-independent (French→English transfer 99.5 % vs 91.5 %). The verdict is written in layers 18–27, mostly by two attention heads and the last MLP; sentence-final periods (2.4 % of tokens) carry 28 % of the evidence and paragraph breaks (0.8 %) 21 %; shuffling words inside sentences removes most of the signal while surface edits barely move it; the score is uncorrelated with a base language model's surprisal.
+
+<img src="assets/interpretability.png" alt="Interpretability" width="100%">
+
+## Training data
+
+Public sources only: RAID, MAGE, COLING-2025 MGT, ai-text-detection-pile, `dmitva/human_ai_generated_text`, Cosmopedia, WildChat, UltraChat, OpenHermes-2.5, arena preference sets, Amazon/Yelp/IMDB reviews, arXiv abstracts, FineWeb-2 (French) and FineWeb-Edu, Aya, Wikipedia, French instruction sets and several public distillation sets from recent frontier models (≈ 1.09 M training documents including augmentations). The dataset builder is in the GitHub repository.
+
+> **Licences.** The weights are released under Apache-2.0 and the backbone is Apache-2.0, but the training data comes from many datasets with their own terms (some research-only or non-commercial). Check them before any commercial use. No dataset is redistributed here.
+
+## Files
+
+`model.safetensors`, `config.json` (merged backbone) · `heads.pt` (the four heads) · `calibrator.json` (decoder calibration, 0.5 % FPR target) · `ouroboros_config.yaml` (inference settings) · tokenizer files.
+
+## Citation
+
+```bibtex
+@software{ouroboros_detector,
+  title  = {Ouroboros: an open token-level AI-text detector},
+  author = {Jourdelune},
+  year   = {2026},
+  note   = {Unofficial reimplementation of the Pangram 4 technical report (arXiv:2607.27183); built end to end with Claude Opus 5.5},
+  url    = {https://github.com/Jourdelune/ouroboros-detector}
+}
+```

calibrator.json

@@ -0,0 +1,41 @@
+{
+  "intercepts": [
+    -0.13786710811109545,
+    0.1736658319082379,
+    -0.03579872379724635
+  ],
+  "weights": [
+    [
+      0.6090387031621195,
+      -0.2203546534333896,
+      0.05042216278611674,
+      -0.043796612923324964,
+      -0.16172733427658842,
+      -0.07141518106119561
+    ],
+    [
+      -0.47584707836701884,
+      0.44584154734933923,
+      -0.53091849183712,
+      0.18571772229692793,
+      0.344271744390476,
+      0.030821937669911974
+    ],
+    [
+      -0.13319162479510294,
+      -0.22548689391594745,
+      0.4804963290510028,
+      -0.1419211093736046,
+      -0.18254441011389036,
+      0.040593243391284464
+    ]
+  ],
+  "deltas": [
+    1.5,
+    0.0,
+    0.0
+  ],
+  "smoothness_lambda": 4.0,
+  "mixed_gamma": 0.25,
+  "identity": false
+}

chat_template.jinja

@@ -0,0 +1,85 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set content = message.content %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is defined and message.reasoning_content is not none %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in message.content %}
+                {%- set content = message.content.split('</think>')[-1].lstrip('\n') %}
+                {%- set reasoning_content = message.content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {%- if loop.last or (not loop.last and reasoning_content) %}
+                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
+            {%- else %}
+                {{- '<|im_start|>' + message.role + '\n' + content }}
+            {%- endif %}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- endif %}
+{%- endif %}

config.json

@@ -0,0 +1,63 @@
+{
+  "architectures": [
+    "Qwen3Model"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "dtype": "bfloat16",
+  "eos_token_id": 151643,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 6144,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 32768,
+  "max_window_layers": 28,
+  "model_type": "qwen3",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 8,
+  "pad_token_id": null,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 1000000,
+    "rope_type": "default"
+  },
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.17.0",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 151936
+}

heads.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f1caa1bcf98884af6aa6edb40e370b767390c4768220ea0e5112a5490ee94df2
+size 199985

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9becd7103e67152a8c8054eabec1dd3a817f6515c4a12dee4576389ec764f8f4
+size 3441183752

ouroboros_config.yaml

@@ -0,0 +1,52 @@
+# Inference / training settings of the released model (Ouroboros v7, stage 2). `backbone.name` is replaced by the
+# directory this file lives in when the model is loaded.
+backbone:
+  name: .
+  dtype: bfloat16
+  attn_implementation: sdpa
+  gradient_checkpointing: true
+  trust_remote_code: false
+lora:
+  r: 32
+  alpha: 64
+  dropout: 0.05
+  target_modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]
+  target_routed_experts: false
+data:
+  train_shards: [data/corpus/train.jsonl]
+  eval_shards: [data/corpus/eval.jsonl]
+  calibration_shards: [data/corpus/calibration.jsonl]
+  test_shards: [data/corpus/test.jsonl]
+  window_tokens: 512
+  stride_tokens: 256
+  min_words: 50
+  max_windows_per_doc: 2
+  mixed_threshold: 0.15
+  soft_bucket_sharpness: 1.0
+  seed: 0
+train:
+  output_dir: runs/ouroboros
+  stage: 2
+  steps: 6000
+  batch_size: 4
+  grad_accum: 4
+  lr: 5.0e-05
+  head_lr: 0.0005
+  weight_decay: 0.01
+  warmup_ratio: 0.03
+  max_grad_norm: 1.0
+  log_every: 25
+  eval_every: 500
+  save_every: 500
+  seed: 0
+  w_segment: 1.0
+  w_token: 1.0
+  w_mixed: 0.5
+  w_humanizer: 0.25
+postprocess:
+  smoothness_lambda: 4.0
+  mixed_gamma: 0.5
+  class_prior_delta: [0.0, 0.0, 0.0]
+  min_segment_tokens: 32
+  human_fraction_threshold: 0.9
+  ai_fraction_threshold: 0.8

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506
+size 11422650

tokenizer_config.json

@@ -0,0 +1,30 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}