Datasets:

openrepro
/

openrepro-ledger

+---
+license: cc-by-4.0
+pretty_name: OpenRepro Ledger
+language:
+- en
+task_categories:
+- other
+tags:
+- reproducibility
+- replication
+- meta-science
+- research-integrity
+- tabular
+size_categories:
+- n<1K
+configs:
+- config_name: ledger
+  data_files:
+  - split: train
+    path: ledger.csv
+  default: true
+- config_name: claims
+  data_files:
+  - split: train
+    path: claims.csv
+---
+# OpenRepro Ledger
+OpenRepro re-runs published computational research and publishes tamper-evident evidence of whether the
+results hold. This dataset is a mirror of the ledger's machine-readable files from
+[openrepro.org](https://openrepro.org/). The site is the source of record. Every value here comes from an
+attempt's `report.json` and is the same as on the site.
+<!-- openrepro:release:start -->
+**Release 2026-10-19.** 7 papers and 53 registered claims (52 held within tolerance). Papers in their author right-of-reply period are not included.
+| arXiv | Paper | Outcome | Headline claims held | Report |
+| --- | --- | --- | --- | --- |
+| [1606.00931](https://arxiv.org/abs/1606.00931) | DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network | Reproduced | 4 of 4 | [report](https://openrepro.org/papers/1606.00931/) |
+| [1607.01759](https://arxiv.org/abs/1607.01759) | Bag of Tricks for Efficient Text Classification | Reproduced | 8 of 8 | [report](https://openrepro.org/papers/1607.01759/) |
+| [1708.05031](https://arxiv.org/abs/1708.05031) | Neural Collaborative Filtering | Reproduced | 4 of 4 | [report](https://openrepro.org/papers/1708.05031/) |
+| [1711.10561](https://arxiv.org/abs/1711.10561) | Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations | Reproduced | 1 of 1 | [report](https://openrepro.org/papers/1711.10561/) |
+| [1810.08473](https://arxiv.org/abs/1810.08473) | From Louvain to Leiden: guaranteeing well-connected communities | Reproduced | 4 of 4 | [report](https://openrepro.org/papers/1810.08473/) |
+| [1812.09970](https://arxiv.org/abs/1812.09970) | Synthetic Difference-in-Differences | Reproduced | 4 of 4 | [report](https://openrepro.org/papers/1812.09970/) |
+| [2004.01105](https://arxiv.org/abs/2004.01105) | Inferring change points in the spread of COVID-19 reveals the effectiveness of interventions | Reproduced | 4 of 4 | [report](https://openrepro.org/papers/2004.01105/) |
+<!-- openrepro:release:end -->
+## Files
+| File | Contents |
+| --- | --- |
+| `ledger.csv` (config `ledger`) | One row per listed paper. |
+| `ledger.json` | The same records under `papers`, plus an envelope: `schema_version`, `version` (the build date), `generated`, `license`, `documentation`, `citation`, `feed`, `mcp`, `papers_listed`. |
+| `claims.csv` (config `claims`) | One row per registered claim of every listed paper. |
+| `claims.json` | The same records under `claims`, with the same envelope. |
+In the CSV files, list fields such as `authors` are joined with `"; "`. Empty cells are nulls.
+```python
+from datasets import load_dataset
+ledger = load_dataset("openrepro/openrepro-ledger", "ledger", split="train")
+claims = load_dataset("openrepro/openrepro-ledger", "claims", split="train")
+```
+### Ledger fields (one record per paper)
+| Field | Meaning |
+| --- | --- |
+| `arxiv_id` | The paper's arXiv id, without a version suffix. It is the stable key for a paper in every file and on the site (`/papers/<arxiv_id>/`). |
+| `title`, `authors`, `year`, `venue`, `doi` | Citation metadata from the report. Display-only corrections are sourced in the site's editorial file. |
+| `field` | A browsing label assigned by OpenRepro. |
+| `outcome`, `outcome_label` | The computed outcome, as a key and as printed: `reproduced`, `reproduced_with_earlier_code`, `partially_reproduced`, `not_reproduced` or `not_attemptable`. |
+| `headline_held`, `headline_held_earlier_code`, `headline_total` | How many headline claims held, how many of those held only with a declared earlier version of the code, and how many headline claims were scored. |
+| `claims_checked` | Every claim registered, headline or not. |
+| `checked` | The date the attempt ended (UTC). |
+| `page_url`, `evidence_url`, `badge_url` | The report page, the full evidence report, and the SVG badge. |
+| `log_head_hash`, `log_events` | The head hash and the number of events in the attempt's hash-chained log. The report page prints this as the seal. |
+| `bundle_url`, `bundle_sha256` | The downloadable evidence bundle and its SHA-256, when published. |
+| `summary` | The report's summary paragraph. |
+### Claims fields (one record per claim)
+| Field | Meaning |
+| --- | --- |
+| `arxiv_id`, `claim_id` | The paper and the claim. Together they are the stable key for a claim; the report page anchor is `#claim-<claim_id>`. |
+| `paper` | The paper's short name. |
+| `headline` | `True` when the claim counts toward the outcome. |
+| `statement`, `location`, `metric`, `unit` | What was registered: the claim in words, where it is in the paper, and what was measured. |
+| `comparison`, `published`, `tolerance`, `accepted_if` | The rule registered before any result was recorded: within a band, at most, or at least; the published value; the tolerance; and the rule as printed. |
+| `observed` | The value obtained by re-running the authors' code. Null when untested. |
+| `verdict` | `consistent`, `consistent_earlier_code`, `inconsistent` or `untested`. |
+| `earlier_code_observed` | The value under the declared earlier-code fallback, when one was tried. |
+| `registered_event`, `first_result_event` | The log events at which the claim was registered and its first result was recorded, so the order can be checked. |
+| `deviation_over_tolerance` | (observed − published) ÷ tolerance. Null when untested; `inf` or `-inf` when the tolerance was zero and the value missed. |
+## How it is produced
+For each paper, an AI agent (Claude, made by Anthropic) works in a sandbox with OpenRepro's engine:
+1. It takes the authors' own code and data.
+2. Before running anything, it registers the claims to check, each with a quote from the paper and a
+   numeric tolerance.
+3. It rebuilds the environment as of the paper's date and re-runs the experiments.
+Every command and output goes into a hash-chained log. The harness computes the verdicts from the registered
+tolerances; the model does not set them. The report is generated from the log. A person oversees the
+project: choosing papers and setting the rules the agent works under. The full method is at
+[openrepro.org/methods/](https://openrepro.org/methods/). The evidence bundles and a one-file verifier are at
+[openrepro.org/verify/](https://openrepro.org/verify/).
+## Limits
+- **AI-run, no expert review.** The re-runs are done by an AI agent. No domain expert has reviewed the
+  reports, and nothing here is peer reviewed.
+- **An outcome is not a judgment of the paper.** It says what happened when the authors' code was re-run under
+  the recorded conditions. "Could not reproduce" means the published artifacts did not produce the published
+  numbers under OpenRepro's protocol. It is not a finding about the authors' honesty or the idea's merit.
+- **Embargoed papers are left out.** For any outcome other than "reproduced", the authors get 14 days to reply
+  before publication. Until that window closes, the paper is absent from these files and from the site.
+- **Small and selective.** The ledger covers only the papers OpenRepro has chosen and finished. It is not a
+  sample of any field and should not be used to estimate a field's reproducibility rate.
+- **Snapshot.** Each version is the ledger as of its build date. A correction is published as a new attempt that
+  links to the old one; ids never change.
+## Update cadence
+The files are rebuilt with the site whenever an attempt finishes and any right-of-reply period has ended. This
+mirror is updated after the site, so it can lag by a few days. `version` in the JSON envelope is the build
+date. The live files are at [openrepro.org/data/](https://openrepro.org/data/).
+## License and citation
+The reports, verdicts, evidence logs and these files are published under
+[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Attribute "OpenRepro, https://openrepro.org" and
+link the report URL so a reader can reach the evidence.
+```bibtex
+@misc{openrepro_ledger,
+  author       = {{OpenRepro}},
+  title        = {{OpenRepro}: a public ledger of re-run computational research},
+  year         = {2026},
+  howpublished = {\url{https://openrepro.org/}},
+  note         = {Dataset, version 2026-10-19. CC BY 4.0. Mirrored at https://huggingface.co/datasets/openrepro/openrepro-ledger}
+}
+```
+The site's [CITATION.cff](https://openrepro.org/CITATION.cff) has the same details. Each report page has its
+own citation block.
+## Contact and right of reply
+Write to [hello@openrepro.org](mailto:hello@openrepro.org) to reply to a report or to report an error. Authors
+may reply at any time, and their notes can be published unedited alongside the report.

claims.csv ADDED Viewed

	@@ -0,0 +1,54 @@

+arxiv_id,paper,claim_id,headline,statement,location,metric,unit,comparison,published,observed,tolerance,accepted_if,verdict,earlier_code_observed,registered_event,first_result_event,deviation_over_tolerance
+1708.05031,Neural Collaborative Filtering,neumf-nopre-8-hr10,True,NeuMF trained from scratch (without pre-training) with 8 predictive factors reaches HR@10 = 0.688 on MovieLens-1M.,"Table 2 (MovieLens rows, Factors = 8, 'Without Pre-training' HR@10 column), p. 7","HR@10 of NeuMF without pre-training, 8 predictive factors, MovieLens-1M, leave-one-out with 99 sampled negatives",ratio,within,0.688,0.6848,0.02,within ±0.02,consistent,,12,38,-0.16
+1708.05031,Neural Collaborative Filtering,neumf-nopre-8-ndcg10,True,NeuMF trained from scratch (without pre-training) with 8 predictive factors reaches NDCG@10 = 0.410 on MovieLens-1M.,"Table 2 (MovieLens rows, Factors = 8, 'Without Pre-training' NDCG@10 column), p. 7","NDCG@10 of NeuMF without pre-training, 8 predictive factors, MovieLens-1M",ratio,within,0.41,0.4088,0.02,within ±0.02,consistent,,13,39,-0.06
+1708.05031,Neural Collaborative Filtering,neumf-pre-8-hr10,True,NeuMF initialised from pre-trained GMF and MLP (alpha = 0.5) with 8 predictive factors reaches HR@10 = 0.684 on MovieLens-1M.,"Table 2 (MovieLens rows, Factors = 8, 'With Pre-training' HR@10 column), p. 7","HR@10 of NeuMF with pre-training, 8 predictive factors, MovieLens-1M",ratio,within,0.684,0.6798,0.02,within ±0.02,consistent,,14,42,-0.21
+1708.05031,Neural Collaborative Filtering,neumf-pre-8-ndcg10,True,NeuMF initialised from pre-trained GMF and MLP (alpha = 0.5) with 8 predictive factors reaches NDCG@10 = 0.403 on MovieLens-1M.,"Table 2 (MovieLens rows, Factors = 8, 'With Pre-training' NDCG@10 column), p. 7","NDCG@10 of NeuMF with pre-training, 8 predictive factors, MovieLens-1M",ratio,within,0.403,0.4054,0.02,within ±0.02,consistent,,15,43,0.12
+1708.05031,Neural Collaborative Filtering,mlp3-8-hr10,False,"MLP with three hidden layers (32->16->8, embedding 16; the README's --layers [64,32,16,8]) reaches HR@10 = 0.671 on MovieLens-1M at 8 predictive factors.","Table 3 (MovieLens rows, Factors = 8, MLP-3 column), p. 8","HR@10 of MLP-3, 8 predictive factors, MovieLens-1M",ratio,within,0.671,0.6707,0.02,within ±0.02,consistent,,16,61,-0.015
+1708.05031,Neural Collaborative Filtering,mlp3-8-ndcg10,False,MLP with three hidden layers reaches NDCG@10 = 0.399 on MovieLens-1M at 8 predictive factors.,"Table 4 (MovieLens rows, Factors = 8, MLP-3 column), p. 9","NDCG@10 of MLP-3, 8 predictive factors, MovieLens-1M",ratio,within,0.399,0.3979,0.02,within ±0.02,consistent,,17,62,-0.055
+1606.00931,DeepSurv,support-cindex,True,DeepSurv reaches test C-index 0.618308 on SUPPORT.,"Table 1, p. 5 (DeepSurv column)",test-set C-index of DeepSurv on SUPPORT,C-index,within,0.618308,0.618582,0.01,within ±0.01,consistent,,23,51,0.0274
+1606.00931,DeepSurv,metabric-cindex,True,DeepSurv reaches test C-index 0.643374 on METABRIC.,"Table 1, p. 5 (DeepSurv column)",test-set C-index of DeepSurv on METABRIC,C-index,within,0.643374,0.643492,0.01,within ±0.01,consistent,,24,52,0.0118
+1606.00931,DeepSurv,gbsg-cindex,True,DeepSurv reaches test C-index 0.668402 on Rotterdam & GBSG.,"Table 1, p. 5 (DeepSurv column)",test-set C-index of DeepSurv on Rotterdam (train) and GBSG (test),C-index,within,0.668402,0.672011,0.01,within ±0.01,consistent,,25,53,0.3609
+1606.00931,DeepSurv,whas-cindex,True,DeepSurv reaches test C-index 0.862620 on WHAS.,"Table 1, p. 5 (DeepSurv column)",test-set C-index of DeepSurv on WHAS,C-index,within,0.86262,0.858003,0.01,within ±0.01,consistent,,26,54,-0.4617
+1606.00931,DeepSurv,sim-linear-cindex,False,DeepSurv reaches test C-index 0.774019 on simulated linear data.,"Table 1, p. 5 (DeepSurv column)",test-set C-index of DeepSurv on the simulated linear survival data,C-index,within,0.774019,0.773885,0.01,within ±0.01,consistent,,27,55,-0.0134
+1606.00931,DeepSurv,sim-nonlinear-cindex,False,DeepSurv reaches test C-index 0.648902 on simulated nonlinear data.,"Table 1, p. 5 (DeepSurv column)",test-set C-index of DeepSurv on the simulated nonlinear (Gaussian) survival data,C-index,within,0.648902,0.645797,0.01,within ±0.01,consistent,,28,56,-0.3105
+1711.10561,Physics Informed Deep Learning (Part I),burgers-l2-order,True,"A physics-informed network (9 layers x 20 neurons, N_u = 100, N_f = 10,000) solves the Burgers equation with relative L2 error of order 1e-4 (reported 6.7e-4).","Section 3.1, Fig. 1 caption, p. 8","relative L2 error between predicted and exact u(t, x), printed by Burgers.py as Error u",relative L2,at_most,0.00067,0.002148728,0.00603,at most 0.0067,consistent,,8,18,0.2452
+1711.10561,Physics Informed Deep Learning (Part I),burgers-l2-close,False,The Burgers relative L2 error is within a factor of 2 of the reported 6.7e-4.,"Section 3.1, Fig. 1 caption, p. 8",relative L2 error printed by Burgers.py as Error u,relative L2,at_most,0.00067,0.002148728,0.00067,at most 0.00134,inconsistent,,9,19,2.2071
+1607.01759,fastText,ag-bigram,True,fastText with 10 hidden units and bigrams reaches 92.5% test accuracy on AG News.,"Table 1, p. 3","test accuracy [%], AG News, fastText h=10 with bigrams",percent,within,92.5,92.2,0.3,within ±0.3,consistent,,10,33,-1.0
+1607.01759,fastText,ag-unigram,False,fastText with 10 hidden units without bigrams reaches 91.5% test accuracy on AG News.,"Table 1, p. 3","test accuracy [%], AG News, fastText h=10 without bigrams",percent,within,91.5,91.4,0.3,within ±0.3,consistent,,11,41,-0.3333
+1607.01759,fastText,sogou-bigram,True,fastText with 10 hidden units and bigrams reaches 96.8% test accuracy on Sogou News.,"Table 1, p. 3","test accuracy [%], Sogou News, fastText h=10 with bigrams",percent,within,96.8,96.7,0.3,within ±0.3,consistent,,12,34,-0.3333
+1607.01759,fastText,sogou-unigram,False,fastText with 10 hidden units without bigrams reaches 93.9% test accuracy on Sogou News.,"Table 1, p. 3","test accuracy [%], Sogou News, fastText h=10 without bigrams",percent,within,93.9,93.7,0.3,within ±0.3,consistent,,13,42,-0.6667
+1607.01759,fastText,dbpedia-bigram,True,fastText with 10 hidden units and bigrams reaches 98.6% test accuracy on DBpedia.,"Table 1, p. 3","test accuracy [%], DBpedia, fastText h=10 with bigrams",percent,within,98.6,98.6,0.3,within ±0.3,consistent,,14,35,0.0
+1607.01759,fastText,dbpedia-unigram,False,fastText with 10 hidden units without bigrams reaches 98.1% test accuracy on DBpedia.,"Table 1, p. 3","test accuracy [%], DBpedia, fastText h=10 without bigrams",percent,within,98.1,98.3,0.3,within ±0.3,consistent,,15,43,0.6667
+1607.01759,fastText,yelp-p-bigram,True,fastText with 10 hidden units and bigrams reaches 95.7% test accuracy on Yelp Polarity.,"Table 1, p. 3","test accuracy [%], Yelp Polarity, fastText h=10 with bigrams",percent,within,95.7,95.7,0.3,within ±0.3,consistent,,16,36,0.0
+1607.01759,fastText,yelp-p-unigram,False,fastText with 10 hidden units without bigrams reaches 93.8% test accuracy on Yelp Polarity.,"Table 1, p. 3","test accuracy [%], Yelp Polarity, fastText h=10 without bigrams",percent,within,93.8,93.9,0.3,within ±0.3,consistent,,17,44,0.3333
+1607.01759,fastText,yelp-f-bigram,True,fastText with 10 hidden units and bigrams reaches 63.9% test accuracy on Yelp Full.,"Table 1, p. 3","test accuracy [%], Yelp Full, fastText h=10 with bigrams",percent,within,63.9,63.9,0.3,within ±0.3,consistent,,18,37,0.0
+1607.01759,fastText,yelp-f-unigram,False,fastText with 10 hidden units without bigrams reaches 60.4% test accuracy on Yelp Full.,"Table 1, p. 3","test accuracy [%], Yelp Full, fastText h=10 without bigrams",percent,within,60.4,60.3,0.3,within ±0.3,consistent,,19,45,-0.3333
+1607.01759,fastText,yahoo-bigram,True,fastText with 10 hidden units and bigrams reaches 72.3% test accuracy on Yahoo! Answers.,"Table 1, p. 3","test accuracy [%], Yahoo! Answers, fastText h=10 with bigrams",percent,within,72.3,72.4,0.3,within ±0.3,consistent,,20,38,0.3333
+1607.01759,fastText,yahoo-unigram,False,fastText with 10 hidden units without bigrams reaches 72.0% test accuracy on Yahoo! Answers.,"Table 1, p. 3","test accuracy [%], Yahoo! Answers, fastText h=10 without bigrams",percent,within,72.0,71.8,0.3,within ±0.3,consistent,,21,46,-0.6667
+1607.01759,fastText,amazon-f-bigram,True,fastText with 10 hidden units and bigrams reaches 60.2% test accuracy on Amazon Full.,"Table 1, p. 3","test accuracy [%], Amazon Full, fastText h=10 with bigrams",percent,within,60.2,60.2,0.3,within ±0.3,consistent,,22,39,0.0
+1607.01759,fastText,amazon-f-unigram,False,fastText with 10 hidden units without bigrams reaches 55.8% test accuracy on Amazon Full.,"Table 1, p. 3","test accuracy [%], Amazon Full, fastText h=10 without bigrams",percent,within,55.8,55.6,0.3,within ±0.3,consistent,,23,47,-0.6667
+1607.01759,fastText,amazon-p-bigram,True,fastText with 10 hidden units and bigrams reaches 94.6% test accuracy on Amazon Polarity.,"Table 1, p. 3","test accuracy [%], Amazon Polarity, fastText h=10 with bigrams",percent,within,94.6,94.6,0.3,within ±0.3,consistent,,24,40,0.0
+1607.01759,fastText,amazon-p-unigram,False,fastText with 10 hidden units without bigrams reaches 91.2% test accuracy on Amazon Polarity.,"Table 1, p. 3","test accuracy [%], Amazon Polarity, fastText h=10 without bigrams",percent,within,91.2,91.1,0.3,within ±0.3,consistent,,25,48,-0.3333
+2004.01105,COVID-19 change points,lambda0,True,Before the first change point the spreading rate in Germany was 0.43 per day.,"Abstract, p. 1","posterior median of lambda_0, three-change-point model, data until 2020-04-21",per day,within,0.43,0.4275,0.02,within ±0.02,consistent,,7,31,-0.125
+2004.01105,COVID-19 change points,lambda1,True,After the first change point the spreading rate was 0.25 per day.,"Abstract, p. 1",posterior median of lambda_1,per day,within,0.25,0.2509,0.02,within ±0.02,consistent,,8,32,0.045
+2004.01105,COVID-19 change points,lambda2,True,After the second change point the spreading rate was 0.15 per day.,"Abstract, p. 1",posterior median of lambda_2,per day,within,0.15,0.1544,0.02,within ±0.02,consistent,,9,33,0.22
+2004.01105,COVID-19 change points,lambda3,True,After the third change point the spreading rate was 0.09 per day.,"Abstract, p. 1",posterior median of lambda_3,per day,within,0.09,0.0937,0.02,within ±0.02,consistent,,10,34,0.185
+2004.01105,COVID-19 change points,cp1-date,False,"The first change point occurred around March 6, 2020.","Abstract, p. 1","posterior median date of change point 1, as day of March 2020",day of March,within,6,5.79,2,within ±2,consistent,,11,35,-0.105
+2004.01105,COVID-19 change points,cp2-date,False,"The second change point occurred around March 15, 2020.","Abstract, p. 1","posterior median date of change point 2, as day of March 2020",day of March,within,15,15.22,2,within ±2,consistent,,12,36,0.11
+2004.01105,COVID-19 change points,cp3-date,False,"The third change point occurred around March 23, 2020.","Abstract, p. 1","posterior median date of change point 3, as day of March 2020",day of March,within,23,22.63,2,within ±2,consistent,,13,37,-0.185
+1810.08473,From Louvain to Leiden,dblp-leiden,True,"On DBLP, the maximal modularity found by the Leiden algorithm over 10 replications of 10 iterations is 0.8387.","Table II, p. 6 (text p. 9)","max modularity (gamma = 1) over 10 random starts x 10 iterations, Leiden, SNAP com-DBLP",modularity,within,0.8387,0.838377,0.002,within ±0.002,consistent,,10,29,-0.1615
+1810.08473,From Louvain to Leiden,amazon-leiden,True,"On Amazon, the maximal modularity found by the Leiden algorithm is 0.9341.","Table II, p. 6 (text p. 9)","max modularity (gamma = 1) over 10 random starts x 10 iterations, Leiden, SNAP com-Amazon",modularity,within,0.9341,0.934022,0.002,within ±0.002,consistent,,11,30,-0.039
+1810.08473,From Louvain to Leiden,dblp-louvain,False,"On DBLP, the maximal modularity found by the Louvain algorithm is 0.8262.","Table II, p. 6 (text p. 9)","max modularity (gamma = 1) over 10 random starts x 10 iterations, Louvain, SNAP com-DBLP",modularity,within,0.8262,0.826653,0.002,within ±0.002,consistent,,12,31,0.2265
+1810.08473,From Louvain to Leiden,amazon-louvain,False,"On Amazon, the maximal modularity found by the Louvain algorithm is 0.9301.","Table II, p. 6 (text p. 9)","max modularity (gamma = 1) over 10 random starts x 10 iterations, Louvain, SNAP com-Amazon",modularity,within,0.9301,0.929846,0.002,within ±0.002,consistent,,13,32,-0.127
+1810.08473,From Louvain to Leiden,dblp-leiden-gain,True,"On DBLP, Leiden finds higher maximal modularity than Louvain (paper: +0.0125).","Table II, p. 6 (text p. 9)",Leiden max modularity minus Louvain max modularity on DBLP,modularity,at_least,0.0125,0.011724,0.0125,at least 0,consistent,,14,33,-0.0621
+1810.08473,From Louvain to Leiden,amazon-leiden-gain,True,"On Amazon, Leiden finds higher maximal modularity than Louvain (paper: +0.0040).","Table II, p. 6 (text p. 9)",Leiden max modularity minus Louvain max modularity on Amazon,modularity,at_least,0.004,0.004176,0.004,at least 0,consistent,,15,34,0.044
+1812.09970,Synthetic Difference-in-Differences,sdid-estimate,True,The SDID estimate of the average effect of Proposition 99 on California per-capita cigarette sales over the 12 post-treatment years is -15.6 packs.,"Table 1, p. 8",SDID point estimate (synthdid_estimate on the California Prop 99 panel),packs per capita,within,-15.6,-15.603828,0.1,within ±0.1,consistent,,9,45,-0.0383
+1812.09970,Synthetic Difference-in-Differences,sdid-se,True,The placebo-method standard error of the SDID estimate is 8.4.,"Table 1, p. 8 (placebo method, Section 5, Algorithm 4)",sqrt of the placebo variance estimate for SDID,packs per capita,within,8.4,8.367993,0.84,within ±0.84,consistent,,10,50,-0.0381
+1812.09970,Synthetic Difference-in-Differences,sc-estimate,True,The synthetic control estimate is -19.6.,"Table 1, p. 8",SC point estimate (sc_estimate),packs per capita,within,-19.6,-19.619663,0.1,within ±0.1,consistent,,11,46,-0.1966
+1812.09970,Synthetic Difference-in-Differences,did-estimate,True,The difference-in-differences estimate is -27.3.,"Table 1, p. 8",DID point estimate (did_estimate),packs per capita,within,-27.3,-27.349111,0.1,within ±0.1,consistent,,12,47,-0.4911
+1812.09970,Synthetic Difference-in-Differences,mc-estimate,False,The matrix completion estimate is -20.2.,"Table 1, p. 8",MC point estimate (MCPanel nuclear-norm estimator as used in the authors code),packs per capita,within,-20.2,-20.218135,0.5,within ±0.5,consistent,,13,48,-0.0363
+1812.09970,Synthetic Difference-in-Differences,difp-estimate,False,The synthetic control with intercept (DIFP) estimate is -11.1.,"Table 1, p. 8",DIFP point estimate,packs per capita,within,-11.1,-11.104645,0.1,within ±0.1,consistent,,14,49,-0.0465
+1812.09970,Synthetic Difference-in-Differences,sc-se,False,The placebo-method standard error of the SC estimate is 9.9.,"Table 1, p. 8",sqrt of the placebo variance estimate for SC,packs per capita,within,9.9,9.917426,0.99,within ±0.99,consistent,,15,51,0.0176
+1812.09970,Synthetic Difference-in-Differences,did-se,False,The placebo-method standard error of the DID estimate is 17.7.,"Table 1, p. 8",sqrt of the placebo variance estimate for DID,packs per capita,within,17.7,17.740267,1.77,within ±1.77,consistent,,16,52,0.0227
+1812.09970,Synthetic Difference-in-Differences,mc-se,False,The placebo-method standard error of the MC estimate is 11.5.,"Table 1, p. 8",sqrt of the placebo variance estimate for MC,packs per capita,within,11.5,11.509864,1.15,within ±1.15,consistent,,17,53,0.0086
+1812.09970,Synthetic Difference-in-Differences,difp-se,False,The placebo-method standard error of the DIFP estimate is 9.5.,"Table 1, p. 8",sqrt of the placebo variance estimate for DIFP,packs per capita,within,9.5,9.45566,0.95,within ±0.95,consistent,,18,54,-0.0467

claims.json ADDED Viewed

	@@ -0,0 +1,1075 @@

+{
+  "schema_version": 1,
+  "version": "2026-10-19",
+  "generated": "2026-10-19",
+  "site": "https://openrepro.org",
+  "license": "https://creativecommons.org/licenses/by/4.0/",
+  "documentation": "https://openrepro.org/data/",
+  "citation": "https://openrepro.org/CITATION.cff",
+  "feed": "https://openrepro.org/feed.xml",
+  "mcp": "https://mcp.openrepro.org/mcp",
+  "papers_listed": 7,
+  "note": "One record per claim of every listed paper; every value comes from the attempt reports. deviation_over_tolerance is (observed − published) ÷ tolerance; null when untested, 'inf' or '-inf' when the tolerance was zero and the value missed.",
+  "claims": [
+    {
+      "arxiv_id": "1708.05031",
+      "paper": "Neural Collaborative Filtering",
+      "claim_id": "neumf-nopre-8-hr10",
+      "headline": true,
+      "statement": "NeuMF trained from scratch (without pre-training) with 8 predictive factors reaches HR@10 = 0.688 on MovieLens-1M.",
+      "location": "Table 2 (MovieLens rows, Factors = 8, 'Without Pre-training' HR@10 column), p. 7",
+      "metric": "HR@10 of NeuMF without pre-training, 8 predictive factors, MovieLens-1M, leave-one-out with 99 sampled negatives",
+      "unit": "ratio",
+      "comparison": "within",
+      "published": 0.688,
+      "observed": 0.6848,
+      "tolerance": 0.02,
+      "accepted_if": "within ±0.02",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 12,
+      "first_result_event": 38,
+      "deviation_over_tolerance": -0.16
+    },
+    {
+      "arxiv_id": "1708.05031",
+      "paper": "Neural Collaborative Filtering",
+      "claim_id": "neumf-nopre-8-ndcg10",
+      "headline": true,
+      "statement": "NeuMF trained from scratch (without pre-training) with 8 predictive factors reaches NDCG@10 = 0.410 on MovieLens-1M.",
+      "location": "Table 2 (MovieLens rows, Factors = 8, 'Without Pre-training' NDCG@10 column), p. 7",
+      "metric": "NDCG@10 of NeuMF without pre-training, 8 predictive factors, MovieLens-1M",
+      "unit": "ratio",
+      "comparison": "within",
+      "published": 0.41,
+      "observed": 0.4088,
+      "tolerance": 0.02,
+      "accepted_if": "within ±0.02",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 13,
+      "first_result_event": 39,
+      "deviation_over_tolerance": -0.06
+    },
+    {
+      "arxiv_id": "1708.05031",
+      "paper": "Neural Collaborative Filtering",
+      "claim_id": "neumf-pre-8-hr10",
+      "headline": true,
+      "statement": "NeuMF initialised from pre-trained GMF and MLP (alpha = 0.5) with 8 predictive factors reaches HR@10 = 0.684 on MovieLens-1M.",
+      "location": "Table 2 (MovieLens rows, Factors = 8, 'With Pre-training' HR@10 column), p. 7",
+      "metric": "HR@10 of NeuMF with pre-training, 8 predictive factors, MovieLens-1M",
+      "unit": "ratio",
+      "comparison": "within",
+      "published": 0.684,
+      "observed": 0.6798,
+      "tolerance": 0.02,
+      "accepted_if": "within ±0.02",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 14,
+      "first_result_event": 42,
+      "deviation_over_tolerance": -0.21
+    },
+    {
+      "arxiv_id": "1708.05031",
+      "paper": "Neural Collaborative Filtering",
+      "claim_id": "neumf-pre-8-ndcg10",
+      "headline": true,
+      "statement": "NeuMF initialised from pre-trained GMF and MLP (alpha = 0.5) with 8 predictive factors reaches NDCG@10 = 0.403 on MovieLens-1M.",
+      "location": "Table 2 (MovieLens rows, Factors = 8, 'With Pre-training' NDCG@10 column), p. 7",
+      "metric": "NDCG@10 of NeuMF with pre-training, 8 predictive factors, MovieLens-1M",
+      "unit": "ratio",
+      "comparison": "within",
+      "published": 0.403,
+      "observed": 0.4054,
+      "tolerance": 0.02,
+      "accepted_if": "within ±0.02",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 15,
+      "first_result_event": 43,
+      "deviation_over_tolerance": 0.12
+    },
+    {
+      "arxiv_id": "1708.05031",
+      "paper": "Neural Collaborative Filtering",
+      "claim_id": "mlp3-8-hr10",
+      "headline": false,
+      "statement": "MLP with three hidden layers (32->16->8, embedding 16; the README's --layers [64,32,16,8]) reaches HR@10 = 0.671 on MovieLens-1M at 8 predictive factors.",
+      "location": "Table 3 (MovieLens rows, Factors = 8, MLP-3 column), p. 8",
+      "metric": "HR@10 of MLP-3, 8 predictive factors, MovieLens-1M",
+      "unit": "ratio",
+      "comparison": "within",
+      "published": 0.671,
+      "observed": 0.6707,
+      "tolerance": 0.02,
+      "accepted_if": "within ±0.02",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 16,
+      "first_result_event": 61,
+      "deviation_over_tolerance": -0.015
+    },
+    {
+      "arxiv_id": "1708.05031",
+      "paper": "Neural Collaborative Filtering",
+      "claim_id": "mlp3-8-ndcg10",
+      "headline": false,
+      "statement": "MLP with three hidden layers reaches NDCG@10 = 0.399 on MovieLens-1M at 8 predictive factors.",
+      "location": "Table 4 (MovieLens rows, Factors = 8, MLP-3 column), p. 9",
+      "metric": "NDCG@10 of MLP-3, 8 predictive factors, MovieLens-1M",
+      "unit": "ratio",
+      "comparison": "within",
+      "published": 0.399,
+      "observed": 0.3979,
+      "tolerance": 0.02,
+      "accepted_if": "within ±0.02",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 17,
+      "first_result_event": 62,
+      "deviation_over_tolerance": -0.055
+    },
+    {
+      "arxiv_id": "1606.00931",
+      "paper": "DeepSurv",
+      "claim_id": "support-cindex",
+      "headline": true,
+      "statement": "DeepSurv reaches test C-index 0.618308 on SUPPORT.",
+      "location": "Table 1, p. 5 (DeepSurv column)",
+      "metric": "test-set C-index of DeepSurv on SUPPORT",
+      "unit": "C-index",
+      "comparison": "within",
+      "published": 0.618308,
+      "observed": 0.618582,
+      "tolerance": 0.01,
+      "accepted_if": "within ±0.01",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 23,
+      "first_result_event": 51,
+      "deviation_over_tolerance": 0.0274
+    },
+    {
+      "arxiv_id": "1606.00931",
+      "paper": "DeepSurv",
+      "claim_id": "metabric-cindex",
+      "headline": true,
+      "statement": "DeepSurv reaches test C-index 0.643374 on METABRIC.",
+      "location": "Table 1, p. 5 (DeepSurv column)",
+      "metric": "test-set C-index of DeepSurv on METABRIC",
+      "unit": "C-index",
+      "comparison": "within",
+      "published": 0.643374,
+      "observed": 0.643492,
+      "tolerance": 0.01,
+      "accepted_if": "within ±0.01",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 24,
+      "first_result_event": 52,
+      "deviation_over_tolerance": 0.0118
+    },
+    {
+      "arxiv_id": "1606.00931",
+      "paper": "DeepSurv",
+      "claim_id": "gbsg-cindex",
+      "headline": true,
+      "statement": "DeepSurv reaches test C-index 0.668402 on Rotterdam & GBSG.",
+      "location": "Table 1, p. 5 (DeepSurv column)",
+      "metric": "test-set C-index of DeepSurv on Rotterdam (train) and GBSG (test)",
+      "unit": "C-index",
+      "comparison": "within",
+      "published": 0.668402,
+      "observed": 0.672011,
+      "tolerance": 0.01,
+      "accepted_if": "within ±0.01",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 25,
+      "first_result_event": 53,
+      "deviation_over_tolerance": 0.3609
+    },
+    {
+      "arxiv_id": "1606.00931",
+      "paper": "DeepSurv",
+      "claim_id": "whas-cindex",
+      "headline": true,
+      "statement": "DeepSurv reaches test C-index 0.862620 on WHAS.",
+      "location": "Table 1, p. 5 (DeepSurv column)",
+      "metric": "test-set C-index of DeepSurv on WHAS",
+      "unit": "C-index",
+      "comparison": "within",
+      "published": 0.86262,
+      "observed": 0.858003,
+      "tolerance": 0.01,
+      "accepted_if": "within ±0.01",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 26,
+      "first_result_event": 54,
+      "deviation_over_tolerance": -0.4617
+    },
+    {
+      "arxiv_id": "1606.00931",
+      "paper": "DeepSurv",
+      "claim_id": "sim-linear-cindex",
+      "headline": false,
+      "statement": "DeepSurv reaches test C-index 0.774019 on simulated linear data.",
+      "location": "Table 1, p. 5 (DeepSurv column)",
+      "metric": "test-set C-index of DeepSurv on the simulated linear survival data",
+      "unit": "C-index",
+      "comparison": "within",
+      "published": 0.774019,
+      "observed": 0.773885,
+      "tolerance": 0.01,
+      "accepted_if": "within ±0.01",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 27,
+      "first_result_event": 55,
+      "deviation_over_tolerance": -0.0134
+    },
+    {
+      "arxiv_id": "1606.00931",
+      "paper": "DeepSurv",
+      "claim_id": "sim-nonlinear-cindex",
+      "headline": false,
+      "statement": "DeepSurv reaches test C-index 0.648902 on simulated nonlinear data.",
+      "location": "Table 1, p. 5 (DeepSurv column)",
+      "metric": "test-set C-index of DeepSurv on the simulated nonlinear (Gaussian) survival data",
+      "unit": "C-index",
+      "comparison": "within",
+      "published": 0.648902,
+      "observed": 0.645797,
+      "tolerance": 0.01,
+      "accepted_if": "within ±0.01",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 28,
+      "first_result_event": 56,
+      "deviation_over_tolerance": -0.3105
+    },
+    {
+      "arxiv_id": "1711.10561",
+      "paper": "Physics Informed Deep Learning (Part I)",
+      "claim_id": "burgers-l2-order",
+      "headline": true,
+      "statement": "A physics-informed network (9 layers x 20 neurons, N_u = 100, N_f = 10,000) solves the Burgers equation with relative L2 error of order 1e-4 (reported 6.7e-4).",
+      "location": "Section 3.1, Fig. 1 caption, p. 8",
+      "metric": "relative L2 error between predicted and exact u(t, x), printed by Burgers.py as Error u",
+      "unit": "relative L2",
+      "comparison": "at_most",
+      "published": 0.00067,
+      "observed": 0.002148728,
+      "tolerance": 0.00603,
+      "accepted_if": "at most 0.0067",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 8,
+      "first_result_event": 18,
+      "deviation_over_tolerance": 0.2452
+    },
+    {
+      "arxiv_id": "1711.10561",
+      "paper": "Physics Informed Deep Learning (Part I)",
+      "claim_id": "burgers-l2-close",
+      "headline": false,
+      "statement": "The Burgers relative L2 error is within a factor of 2 of the reported 6.7e-4.",
+      "location": "Section 3.1, Fig. 1 caption, p. 8",
+      "metric": "relative L2 error printed by Burgers.py as Error u",
+      "unit": "relative L2",
+      "comparison": "at_most",
+      "published": 0.00067,
+      "observed": 0.002148728,
+      "tolerance": 0.00067,
+      "accepted_if": "at most 0.00134",
+      "verdict": "inconsistent",
+      "earlier_code_observed": null,
+      "registered_event": 9,
+      "first_result_event": 19,
+      "deviation_over_tolerance": 2.2071
+    },
+    {
+      "arxiv_id": "1607.01759",
+      "paper": "fastText",
+      "claim_id": "ag-bigram",
+      "headline": true,
+      "statement": "fastText with 10 hidden units and bigrams reaches 92.5% test accuracy on AG News.",
+      "location": "Table 1, p. 3",
+      "metric": "test accuracy [%], AG News, fastText h=10 with bigrams",
+      "unit": "percent",
+      "comparison": "within",
+      "published": 92.5,
+      "observed": 92.2,
+      "tolerance": 0.3,
+      "accepted_if": "within ±0.3",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 10,
+      "first_result_event": 33,
+      "deviation_over_tolerance": -1.0
+    },
+    {
+      "arxiv_id": "1607.01759",
+      "paper": "fastText",
+      "claim_id": "ag-unigram",
+      "headline": false,
+      "statement": "fastText with 10 hidden units without bigrams reaches 91.5% test accuracy on AG News.",
+      "location": "Table 1, p. 3",
+      "metric": "test accuracy [%], AG News, fastText h=10 without bigrams",
+      "unit": "percent",
+      "comparison": "within",
+      "published": 91.5,
+      "observed": 91.4,
+      "tolerance": 0.3,
+      "accepted_if": "within ±0.3",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 11,
+      "first_result_event": 41,
+      "deviation_over_tolerance": -0.3333
+    },
+    {
+      "arxiv_id": "1607.01759",
+      "paper": "fastText",
+      "claim_id": "sogou-bigram",
+      "headline": true,
+      "statement": "fastText with 10 hidden units and bigrams reaches 96.8% test accuracy on Sogou News.",
+      "location": "Table 1, p. 3",
+      "metric": "test accuracy [%], Sogou News, fastText h=10 with bigrams",
+      "unit": "percent",
+      "comparison": "within",
+      "published": 96.8,
+      "observed": 96.7,
+      "tolerance": 0.3,
+      "accepted_if": "within ±0.3",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 12,
+      "first_result_event": 34,
+      "deviation_over_tolerance": -0.3333
+    },
+    {
+      "arxiv_id": "1607.01759",
+      "paper": "fastText",
+      "claim_id": "sogou-unigram",
+      "headline": false,
+      "statement": "fastText with 10 hidden units without bigrams reaches 93.9% test accuracy on Sogou News.",
+      "location": "Table 1, p. 3",
+      "metric": "test accuracy [%], Sogou News, fastText h=10 without bigrams",
+      "unit": "percent",
+      "comparison": "within",
+      "published": 93.9,
+      "observed": 93.7,
+      "tolerance": 0.3,
+      "accepted_if": "within ±0.3",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 13,
+      "first_result_event": 42,
+      "deviation_over_tolerance": -0.6667
+    },
+    {
+      "arxiv_id": "1607.01759",
+      "paper": "fastText",
+      "claim_id": "dbpedia-bigram",
+      "headline": true,
+      "statement": "fastText with 10 hidden units and bigrams reaches 98.6% test accuracy on DBpedia.",
+      "location": "Table 1, p. 3",
+      "metric": "test accuracy [%], DBpedia, fastText h=10 with bigrams",
+      "unit": "percent",
+      "comparison": "within",
+      "published": 98.6,
+      "observed": 98.6,
+      "tolerance": 0.3,
+      "accepted_if": "within ±0.3",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 14,
+      "first_result_event": 35,
+      "deviation_over_tolerance": 0.0
+    },
+    {
+      "arxiv_id": "1607.01759",
+      "paper": "fastText",
+      "claim_id": "dbpedia-unigram",
+      "headline": false,
+      "statement": "fastText with 10 hidden units without bigrams reaches 98.1% test accuracy on DBpedia.",
+      "location": "Table 1, p. 3",
+      "metric": "test accuracy [%], DBpedia, fastText h=10 without bigrams",
+      "unit": "percent",
+      "comparison": "within",
+      "published": 98.1,
+      "observed": 98.3,
+      "tolerance": 0.3,
+      "accepted_if": "within ±0.3",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 15,
+      "first_result_event": 43,
+      "deviation_over_tolerance": 0.6667
+    },
+    {
+      "arxiv_id": "1607.01759",
+      "paper": "fastText",
+      "claim_id": "yelp-p-bigram",
+      "headline": true,
+      "statement": "fastText with 10 hidden units and bigrams reaches 95.7% test accuracy on Yelp Polarity.",
+      "location": "Table 1, p. 3",
+      "metric": "test accuracy [%], Yelp Polarity, fastText h=10 with bigrams",
+      "unit": "percent",
+      "comparison": "within",
+      "published": 95.7,
+      "observed": 95.7,
+      "tolerance": 0.3,
+      "accepted_if": "within ±0.3",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 16,
+      "first_result_event": 36,
+      "deviation_over_tolerance": 0.0
+    },
+    {
+      "arxiv_id": "1607.01759",
+      "paper": "fastText",
+      "claim_id": "yelp-p-unigram",
+      "headline": false,
+      "statement": "fastText with 10 hidden units without bigrams reaches 93.8% test accuracy on Yelp Polarity.",
+      "location": "Table 1, p. 3",
+      "metric": "test accuracy [%], Yelp Polarity, fastText h=10 without bigrams",
+      "unit": "percent",
+      "comparison": "within",
+      "published": 93.8,
+      "observed": 93.9,
+      "tolerance": 0.3,
+      "accepted_if": "within ±0.3",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 17,
+      "first_result_event": 44,
+      "deviation_over_tolerance": 0.3333
+    },
+    {
+      "arxiv_id": "1607.01759",
+      "paper": "fastText",
+      "claim_id": "yelp-f-bigram",
+      "headline": true,
+      "statement": "fastText with 10 hidden units and bigrams reaches 63.9% test accuracy on Yelp Full.",
+      "location": "Table 1, p. 3",
+      "metric": "test accuracy [%], Yelp Full, fastText h=10 with bigrams",
+      "unit": "percent",
+      "comparison": "within",
+      "published": 63.9,
+      "observed": 63.9,
+      "tolerance": 0.3,
+      "accepted_if": "within ±0.3",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 18,
+      "first_result_event": 37,
+      "deviation_over_tolerance": 0.0
+    },
+    {
+      "arxiv_id": "1607.01759",
+      "paper": "fastText",
+      "claim_id": "yelp-f-unigram",
+      "headline": false,
+      "statement": "fastText with 10 hidden units without bigrams reaches 60.4% test accuracy on Yelp Full.",
+      "location": "Table 1, p. 3",
+      "metric": "test accuracy [%], Yelp Full, fastText h=10 without bigrams",
+      "unit": "percent",
+      "comparison": "within",
+      "published": 60.4,
+      "observed": 60.3,
+      "tolerance": 0.3,
+      "accepted_if": "within ±0.3",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 19,
+      "first_result_event": 45,
+      "deviation_over_tolerance": -0.3333
+    },
+    {
+      "arxiv_id": "1607.01759",
+      "paper": "fastText",
+      "claim_id": "yahoo-bigram",
+      "headline": true,
+      "statement": "fastText with 10 hidden units and bigrams reaches 72.3% test accuracy on Yahoo! Answers.",
+      "location": "Table 1, p. 3",
+      "metric": "test accuracy [%], Yahoo! Answers, fastText h=10 with bigrams",
+      "unit": "percent",
+      "comparison": "within",
+      "published": 72.3,
+      "observed": 72.4,
+      "tolerance": 0.3,
+      "accepted_if": "within ±0.3",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 20,
+      "first_result_event": 38,
+      "deviation_over_tolerance": 0.3333
+    },
+    {
+      "arxiv_id": "1607.01759",
+      "paper": "fastText",
+      "claim_id": "yahoo-unigram",
+      "headline": false,
+      "statement": "fastText with 10 hidden units without bigrams reaches 72.0% test accuracy on Yahoo! Answers.",
+      "location": "Table 1, p. 3",
+      "metric": "test accuracy [%], Yahoo! Answers, fastText h=10 without bigrams",
+      "unit": "percent",
+      "comparison": "within",
+      "published": 72.0,
+      "observed": 71.8,
+      "tolerance": 0.3,
+      "accepted_if": "within ±0.3",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 21,
+      "first_result_event": 46,
+      "deviation_over_tolerance": -0.6667
+    },
+    {
+      "arxiv_id": "1607.01759",
+      "paper": "fastText",
+      "claim_id": "amazon-f-bigram",
+      "headline": true,
+      "statement": "fastText with 10 hidden units and bigrams reaches 60.2% test accuracy on Amazon Full.",
+      "location": "Table 1, p. 3",
+      "metric": "test accuracy [%], Amazon Full, fastText h=10 with bigrams",
+      "unit": "percent",
+      "comparison": "within",
+      "published": 60.2,
+      "observed": 60.2,
+      "tolerance": 0.3,
+      "accepted_if": "within ±0.3",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 22,
+      "first_result_event": 39,
+      "deviation_over_tolerance": 0.0
+    },
+    {
+      "arxiv_id": "1607.01759",
+      "paper": "fastText",
+      "claim_id": "amazon-f-unigram",
+      "headline": false,
+      "statement": "fastText with 10 hidden units without bigrams reaches 55.8% test accuracy on Amazon Full.",
+      "location": "Table 1, p. 3",
+      "metric": "test accuracy [%], Amazon Full, fastText h=10 without bigrams",
+      "unit": "percent",
+      "comparison": "within",
+      "published": 55.8,
+      "observed": 55.6,
+      "tolerance": 0.3,
+      "accepted_if": "within ±0.3",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 23,
+      "first_result_event": 47,
+      "deviation_over_tolerance": -0.6667
+    },
+    {
+      "arxiv_id": "1607.01759",
+      "paper": "fastText",
+      "claim_id": "amazon-p-bigram",
+      "headline": true,
+      "statement": "fastText with 10 hidden units and bigrams reaches 94.6% test accuracy on Amazon Polarity.",
+      "location": "Table 1, p. 3",
+      "metric": "test accuracy [%], Amazon Polarity, fastText h=10 with bigrams",
+      "unit": "percent",
+      "comparison": "within",
+      "published": 94.6,
+      "observed": 94.6,
+      "tolerance": 0.3,
+      "accepted_if": "within ±0.3",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 24,
+      "first_result_event": 40,
+      "deviation_over_tolerance": 0.0
+    },
+    {
+      "arxiv_id": "1607.01759",
+      "paper": "fastText",
+      "claim_id": "amazon-p-unigram",
+      "headline": false,
+      "statement": "fastText with 10 hidden units without bigrams reaches 91.2% test accuracy on Amazon Polarity.",
+      "location": "Table 1, p. 3",
+      "metric": "test accuracy [%], Amazon Polarity, fastText h=10 without bigrams",
+      "unit": "percent",
+      "comparison": "within",
+      "published": 91.2,
+      "observed": 91.1,
+      "tolerance": 0.3,
+      "accepted_if": "within ±0.3",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 25,
+      "first_result_event": 48,
+      "deviation_over_tolerance": -0.3333
+    },
+    {
+      "arxiv_id": "2004.01105",
+      "paper": "COVID-19 change points",
+      "claim_id": "lambda0",
+      "headline": true,
+      "statement": "Before the first change point the spreading rate in Germany was 0.43 per day.",
+      "location": "Abstract, p. 1",
+      "metric": "posterior median of lambda_0, three-change-point model, data until 2020-04-21",
+      "unit": "per day",
+      "comparison": "within",
+      "published": 0.43,
+      "observed": 0.4275,
+      "tolerance": 0.02,
+      "accepted_if": "within ±0.02",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 7,
+      "first_result_event": 31,
+      "deviation_over_tolerance": -0.125
+    },
+    {
+      "arxiv_id": "2004.01105",
+      "paper": "COVID-19 change points",
+      "claim_id": "lambda1",
+      "headline": true,
+      "statement": "After the first change point the spreading rate was 0.25 per day.",
+      "location": "Abstract, p. 1",
+      "metric": "posterior median of lambda_1",
+      "unit": "per day",
+      "comparison": "within",
+      "published": 0.25,
+      "observed": 0.2509,
+      "tolerance": 0.02,
+      "accepted_if": "within ±0.02",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 8,
+      "first_result_event": 32,
+      "deviation_over_tolerance": 0.045
+    },
+    {
+      "arxiv_id": "2004.01105",
+      "paper": "COVID-19 change points",
+      "claim_id": "lambda2",
+      "headline": true,
+      "statement": "After the second change point the spreading rate was 0.15 per day.",
+      "location": "Abstract, p. 1",
+      "metric": "posterior median of lambda_2",
+      "unit": "per day",
+      "comparison": "within",
+      "published": 0.15,
+      "observed": 0.1544,
+      "tolerance": 0.02,
+      "accepted_if": "within ±0.02",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 9,
+      "first_result_event": 33,
+      "deviation_over_tolerance": 0.22
+    },
+    {
+      "arxiv_id": "2004.01105",
+      "paper": "COVID-19 change points",
+      "claim_id": "lambda3",
+      "headline": true,
+      "statement": "After the third change point the spreading rate was 0.09 per day.",
+      "location": "Abstract, p. 1",
+      "metric": "posterior median of lambda_3",
+      "unit": "per day",
+      "comparison": "within",
+      "published": 0.09,
+      "observed": 0.0937,
+      "tolerance": 0.02,
+      "accepted_if": "within ±0.02",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 10,
+      "first_result_event": 34,
+      "deviation_over_tolerance": 0.185
+    },
+    {
+      "arxiv_id": "2004.01105",
+      "paper": "COVID-19 change points",
+      "claim_id": "cp1-date",
+      "headline": false,
+      "statement": "The first change point occurred around March 6, 2020.",
+      "location": "Abstract, p. 1",
+      "metric": "posterior median date of change point 1, as day of March 2020",
+      "unit": "day of March",
+      "comparison": "within",
+      "published": 6,
+      "observed": 5.79,
+      "tolerance": 2,
+      "accepted_if": "within ±2",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 11,
+      "first_result_event": 35,
+      "deviation_over_tolerance": -0.105
+    },
+    {
+      "arxiv_id": "2004.01105",
+      "paper": "COVID-19 change points",
+      "claim_id": "cp2-date",
+      "headline": false,
+      "statement": "The second change point occurred around March 15, 2020.",
+      "location": "Abstract, p. 1",
+      "metric": "posterior median date of change point 2, as day of March 2020",
+      "unit": "day of March",
+      "comparison": "within",
+      "published": 15,
+      "observed": 15.22,
+      "tolerance": 2,
+      "accepted_if": "within ±2",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 12,
+      "first_result_event": 36,
+      "deviation_over_tolerance": 0.11
+    },
+    {
+      "arxiv_id": "2004.01105",
+      "paper": "COVID-19 change points",
+      "claim_id": "cp3-date",
+      "headline": false,
+      "statement": "The third change point occurred around March 23, 2020.",
+      "location": "Abstract, p. 1",
+      "metric": "posterior median date of change point 3, as day of March 2020",
+      "unit": "day of March",
+      "comparison": "within",
+      "published": 23,
+      "observed": 22.63,
+      "tolerance": 2,
+      "accepted_if": "within ±2",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 13,
+      "first_result_event": 37,
+      "deviation_over_tolerance": -0.185
+    },
+    {
+      "arxiv_id": "1810.08473",
+      "paper": "From Louvain to Leiden",
+      "claim_id": "dblp-leiden",
+      "headline": true,
+      "statement": "On DBLP, the maximal modularity found by the Leiden algorithm over 10 replications of 10 iterations is 0.8387.",
+      "location": "Table II, p. 6 (text p. 9)",
+      "metric": "max modularity (gamma = 1) over 10 random starts x 10 iterations, Leiden, SNAP com-DBLP",
+      "unit": "modularity",
+      "comparison": "within",
+      "published": 0.8387,
+      "observed": 0.838377,
+      "tolerance": 0.002,
+      "accepted_if": "within ±0.002",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 10,
+      "first_result_event": 29,
+      "deviation_over_tolerance": -0.1615
+    },
+    {
+      "arxiv_id": "1810.08473",
+      "paper": "From Louvain to Leiden",
+      "claim_id": "amazon-leiden",
+      "headline": true,
+      "statement": "On Amazon, the maximal modularity found by the Leiden algorithm is 0.9341.",
+      "location": "Table II, p. 6 (text p. 9)",
+      "metric": "max modularity (gamma = 1) over 10 random starts x 10 iterations, Leiden, SNAP com-Amazon",
+      "unit": "modularity",
+      "comparison": "within",
+      "published": 0.9341,
+      "observed": 0.934022,
+      "tolerance": 0.002,
+      "accepted_if": "within ±0.002",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 11,
+      "first_result_event": 30,
+      "deviation_over_tolerance": -0.039
+    },
+    {
+      "arxiv_id": "1810.08473",
+      "paper": "From Louvain to Leiden",
+      "claim_id": "dblp-louvain",
+      "headline": false,
+      "statement": "On DBLP, the maximal modularity found by the Louvain algorithm is 0.8262.",
+      "location": "Table II, p. 6 (text p. 9)",
+      "metric": "max modularity (gamma = 1) over 10 random starts x 10 iterations, Louvain, SNAP com-DBLP",
+      "unit": "modularity",
+      "comparison": "within",
+      "published": 0.8262,
+      "observed": 0.826653,
+      "tolerance": 0.002,
+      "accepted_if": "within ±0.002",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 12,
+      "first_result_event": 31,
+      "deviation_over_tolerance": 0.2265
+    },
+    {
+      "arxiv_id": "1810.08473",
+      "paper": "From Louvain to Leiden",
+      "claim_id": "amazon-louvain",
+      "headline": false,
+      "statement": "On Amazon, the maximal modularity found by the Louvain algorithm is 0.9301.",
+      "location": "Table II, p. 6 (text p. 9)",
+      "metric": "max modularity (gamma = 1) over 10 random starts x 10 iterations, Louvain, SNAP com-Amazon",
+      "unit": "modularity",
+      "comparison": "within",
+      "published": 0.9301,
+      "observed": 0.929846,
+      "tolerance": 0.002,
+      "accepted_if": "within ±0.002",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 13,
+      "first_result_event": 32,
+      "deviation_over_tolerance": -0.127
+    },
+    {
+      "arxiv_id": "1810.08473",
+      "paper": "From Louvain to Leiden",
+      "claim_id": "dblp-leiden-gain",
+      "headline": true,
+      "statement": "On DBLP, Leiden finds higher maximal modularity than Louvain (paper: +0.0125).",
+      "location": "Table II, p. 6 (text p. 9)",
+      "metric": "Leiden max modularity minus Louvain max modularity on DBLP",
+      "unit": "modularity",
+      "comparison": "at_least",
+      "published": 0.0125,
+      "observed": 0.011724,
+      "tolerance": 0.0125,
+      "accepted_if": "at least 0",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 14,
+      "first_result_event": 33,
+      "deviation_over_tolerance": -0.0621
+    },
+    {
+      "arxiv_id": "1810.08473",
+      "paper": "From Louvain to Leiden",
+      "claim_id": "amazon-leiden-gain",
+      "headline": true,
+      "statement": "On Amazon, Leiden finds higher maximal modularity than Louvain (paper: +0.0040).",
+      "location": "Table II, p. 6 (text p. 9)",
+      "metric": "Leiden max modularity minus Louvain max modularity on Amazon",
+      "unit": "modularity",
+      "comparison": "at_least",
+      "published": 0.004,
+      "observed": 0.004176,
+      "tolerance": 0.004,
+      "accepted_if": "at least 0",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 15,
+      "first_result_event": 34,
+      "deviation_over_tolerance": 0.044
+    },
+    {
+      "arxiv_id": "1812.09970",
+      "paper": "Synthetic Difference-in-Differences",
+      "claim_id": "sdid-estimate",
+      "headline": true,
+      "statement": "The SDID estimate of the average effect of Proposition 99 on California per-capita cigarette sales over the 12 post-treatment years is -15.6 packs.",
+      "location": "Table 1, p. 8",
+      "metric": "SDID point estimate (synthdid_estimate on the California Prop 99 panel)",
+      "unit": "packs per capita",
+      "comparison": "within",
+      "published": -15.6,
+      "observed": -15.603828,
+      "tolerance": 0.1,
+      "accepted_if": "within ±0.1",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 9,
+      "first_result_event": 45,
+      "deviation_over_tolerance": -0.0383
+    },
+    {
+      "arxiv_id": "1812.09970",
+      "paper": "Synthetic Difference-in-Differences",
+      "claim_id": "sdid-se",
+      "headline": true,
+      "statement": "The placebo-method standard error of the SDID estimate is 8.4.",
+      "location": "Table 1, p. 8 (placebo method, Section 5, Algorithm 4)",
+      "metric": "sqrt of the placebo variance estimate for SDID",
+      "unit": "packs per capita",
+      "comparison": "within",
+      "published": 8.4,
+      "observed": 8.367993,
+      "tolerance": 0.84,
+      "accepted_if": "within ±0.84",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 10,
+      "first_result_event": 50,
+      "deviation_over_tolerance": -0.0381
+    },
+    {
+      "arxiv_id": "1812.09970",
+      "paper": "Synthetic Difference-in-Differences",
+      "claim_id": "sc-estimate",
+      "headline": true,
+      "statement": "The synthetic control estimate is -19.6.",
+      "location": "Table 1, p. 8",
+      "metric": "SC point estimate (sc_estimate)",
+      "unit": "packs per capita",
+      "comparison": "within",
+      "published": -19.6,
+      "observed": -19.619663,
+      "tolerance": 0.1,
+      "accepted_if": "within ±0.1",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 11,
+      "first_result_event": 46,
+      "deviation_over_tolerance": -0.1966
+    },
+    {
+      "arxiv_id": "1812.09970",
+      "paper": "Synthetic Difference-in-Differences",
+      "claim_id": "did-estimate",
+      "headline": true,
+      "statement": "The difference-in-differences estimate is -27.3.",
+      "location": "Table 1, p. 8",
+      "metric": "DID point estimate (did_estimate)",
+      "unit": "packs per capita",
+      "comparison": "within",
+      "published": -27.3,
+      "observed": -27.349111,
+      "tolerance": 0.1,
+      "accepted_if": "within ±0.1",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 12,
+      "first_result_event": 47,
+      "deviation_over_tolerance": -0.4911
+    },
+    {
+      "arxiv_id": "1812.09970",
+      "paper": "Synthetic Difference-in-Differences",
+      "claim_id": "mc-estimate",
+      "headline": false,
+      "statement": "The matrix completion estimate is -20.2.",
+      "location": "Table 1, p. 8",
+      "metric": "MC point estimate (MCPanel nuclear-norm estimator as used in the authors code)",
+      "unit": "packs per capita",
+      "comparison": "within",
+      "published": -20.2,
+      "observed": -20.218135,
+      "tolerance": 0.5,
+      "accepted_if": "within ±0.5",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 13,
+      "first_result_event": 48,
+      "deviation_over_tolerance": -0.0363
+    },
+    {
+      "arxiv_id": "1812.09970",
+      "paper": "Synthetic Difference-in-Differences",
+      "claim_id": "difp-estimate",
+      "headline": false,
+      "statement": "The synthetic control with intercept (DIFP) estimate is -11.1.",
+      "location": "Table 1, p. 8",
+      "metric": "DIFP point estimate",
+      "unit": "packs per capita",
+      "comparison": "within",
+      "published": -11.1,
+      "observed": -11.104645,
+      "tolerance": 0.1,
+      "accepted_if": "within ±0.1",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 14,
+      "first_result_event": 49,
+      "deviation_over_tolerance": -0.0465
+    },
+    {
+      "arxiv_id": "1812.09970",
+      "paper": "Synthetic Difference-in-Differences",
+      "claim_id": "sc-se",
+      "headline": false,
+      "statement": "The placebo-method standard error of the SC estimate is 9.9.",
+      "location": "Table 1, p. 8",
+      "metric": "sqrt of the placebo variance estimate for SC",
+      "unit": "packs per capita",
+      "comparison": "within",
+      "published": 9.9,
+      "observed": 9.917426,
+      "tolerance": 0.99,
+      "accepted_if": "within ±0.99",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 15,
+      "first_result_event": 51,
+      "deviation_over_tolerance": 0.0176
+    },
+    {
+      "arxiv_id": "1812.09970",
+      "paper": "Synthetic Difference-in-Differences",
+      "claim_id": "did-se",
+      "headline": false,
+      "statement": "The placebo-method standard error of the DID estimate is 17.7.",
+      "location": "Table 1, p. 8",
+      "metric": "sqrt of the placebo variance estimate for DID",
+      "unit": "packs per capita",
+      "comparison": "within",
+      "published": 17.7,
+      "observed": 17.740267,
+      "tolerance": 1.77,
+      "accepted_if": "within ±1.77",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 16,
+      "first_result_event": 52,
+      "deviation_over_tolerance": 0.0227
+    },
+    {
+      "arxiv_id": "1812.09970",
+      "paper": "Synthetic Difference-in-Differences",
+      "claim_id": "mc-se",
+      "headline": false,
+      "statement": "The placebo-method standard error of the MC estimate is 11.5.",
+      "location": "Table 1, p. 8",
+      "metric": "sqrt of the placebo variance estimate for MC",
+      "unit": "packs per capita",
+      "comparison": "within",
+      "published": 11.5,
+      "observed": 11.509864,
+      "tolerance": 1.15,
+      "accepted_if": "within ±1.15",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 17,
+      "first_result_event": 53,
+      "deviation_over_tolerance": 0.0086
+    },
+    {
+      "arxiv_id": "1812.09970",
+      "paper": "Synthetic Difference-in-Differences",
+      "claim_id": "difp-se",
+      "headline": false,
+      "statement": "The placebo-method standard error of the DIFP estimate is 9.5.",
+      "location": "Table 1, p. 8",
+      "metric": "sqrt of the placebo variance estimate for DIFP",
+      "unit": "packs per capita",
+      "comparison": "within",
+      "published": 9.5,
+      "observed": 9.45566,
+      "tolerance": 0.95,
+      "accepted_if": "within ±0.95",
+      "verdict": "consistent",
+      "earlier_code_observed": null,
+      "registered_event": 18,
+      "first_result_event": 54,
+      "deviation_over_tolerance": -0.0467
+    }
+  ]
+}

ledger.csv ADDED Viewed

	@@ -0,0 +1,8 @@

+arxiv_id,title,authors,year,venue,doi,field,outcome,outcome_label,headline_held,headline_held_earlier_code,headline_total,claims_checked,checked,page_url,evidence_url,badge_url,log_head_hash,log_events,bundle_url,bundle_sha256,summary
+1708.05031,Neural Collaborative Filtering,Xiangnan He; Lizi Liao; Hanwang Zhang; Liqiang Nie; Xia Hu; Tat-Seng Chua,2017,,,Recommender systems,reproduced,Reproduced,4,0,4,6,2026-10-19,https://openrepro.org/papers/1708.05031/,https://openrepro.org/evidence/arxiv-1708-05031-20261019T135218Z.html,https://openrepro.org/badge/1708.05031.svg,04ac3824830007550ce23545b33c04ec2f8c290dd7107cca03355bd8ce4ddfc6,67,https://openrepro.org/bundles/arxiv-1708-05031-20261019T135218Z.tar.gz,3e70c77dc0f0de0d0be0fbe4eeb0b7b66ebb735a00446ace8a237b3426bfcd99,"The authors Keras code (commit 312aff0, identical in code to the current default branch) was run on MovieLens-1M with the exact README environment (Python 2.7.15, Keras 1.0.7, Theano 0.8.0) on CPU, using the README commands with 8 predictive factors and 20 epochs. All six pre-registered Table 2/3/4 values at 8 factors landed within 0.005 of the paper: NeuMF without pre-training HR@10 0.6848 vs 0.688 and NDCG@10 0.4088 vs 0.410; NeuMF with pre-training (initialised from the authors bundled GMF/MLP weights rather than retrained ones) 0.6798 vs 0.684 and 0.4054 vs 0.403; MLP-3 0.6707 vs 0.671 and 0.3979 vs 0.399. Each value is one run reported at the best-HR epoch, which is the scripts own convention. Mid-attempt the shared cloud droplet was destroyed by another process; the environment and code were rebuilt with the same recipe and only the in-flight MLP run was repeated."
+1606.00931,DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network,Jared Katzman; Uri Shaham; Alexander Cloninger; Jonathan Bates; Tingting Jiang; Yuval Kluger,2018,BMC Medical Research Methodology,10.1186/s12874-018-0482-1,Medical statistics,reproduced,Reproduced,4,0,4,6,2026-10-19,https://openrepro.org/papers/1606.00931/,https://openrepro.org/evidence/arxiv-1606-00931-20261018T225505Z.html,https://openrepro.org/badge/1606.00931.svg,b6c148f9a6936f84ea439e56d1c64f0a7dcb49e564e06bcc17ae75d7dd7f225f,63,https://openrepro.org/bundles/arxiv-1606-00931-20261018T225505Z.tar.gz,9b7162f05b98080f2e77db8052c1917f05bd67a5dac3eb3eff03b43872d17cb4,"Rerunning the first author code at the commit matching arXiv v3 (079dee7, 2017-08-08), with the documented command for each experiment in a reconstructed Python 2.7 / Theano 0.9 / Lasagne environment on CPU, reproduces all six DeepSurv C-index values of Table 1 within 0.01: mean of 10 unseeded runs SUPPORT 0.6186 vs 0.6183, METABRIC 0.6435 vs 0.6434, Rotterdam & GBSG 0.6720 vs 0.6684, WHAS 0.8580 vs 0.8626 (run-to-run sd 0.009), simulated linear 0.7739 vs 0.7740, simulated nonlinear 0.6458 vs 0.6489. The WHAS model file in the repository does not match the WHAS hyper-parameters listed in the paper Table 3, and every run ends with a weight-saving error after the metrics are printed (fixed upstream in December 2017). CPH, RSF and the treatment-recommendation results were not scored."
+1711.10561,Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations,Maziar Raissi; Paris G. Perdikaris; George Em Karniadakis,2017,,,Scientific machine learning,reproduced,Reproduced,1,0,1,2,2026-10-19,https://openrepro.org/papers/1711.10561/,https://openrepro.org/evidence/arxiv-1711-10561-20261018T234638Z.html,https://openrepro.org/badge/1711.10561.svg,450dad5c4edbfc2ab0c9dce568dcddd793d19978beba85e4a64f26f6724759da,26,https://openrepro.org/bundles/arxiv-1711-10561-20261018T234638Z.tar.gz,888f02774eeceb719c5891f24233a60b9a06bec62af60dd927f93a94d24d838c,"Running the authors Burgers.py (the earliest version in their repository, from five months after arXiv v1) with its fixed seeds on CPU gives a relative L2 error of 2.15e-3 for the continuous-time Burgers example, against 6.7e-4 reported. That meets the pre-registered order-of-magnitude headline check (at most 6.7e-3) but not the stricter within-a-factor-of-2 check. The paper own Table 1 shows errors varying between 1e-4 and 1e-2 across neighbouring settings, and this run used a CPU rather than the reported GPU. The Table 1 sweep was not run."
+1607.01759,Bag of Tricks for Efficient Text Classification,Armand Joulin; Édouard Grave; Piotr Bojanowski; Tomáš Mikolov,2017,Conference of the European Chapter of the Association for Computational Linguistics,10.18653/v1/e17-2068,Natural language processing,reproduced,Reproduced,8,0,8,16,2026-10-18,https://openrepro.org/papers/1607.01759/,https://openrepro.org/evidence/arxiv-1607-01759-20261018T223713Z.html,https://openrepro.org/badge/1607.01759.svg,87b5ac6197ec5f210cb74ba6aaa5412fe8e2237bdf845f1afee95be9043de61d,56,https://openrepro.org/bundles/arxiv-1607-01759-20261018T223713Z.tar.gz,2ed5e509997e43ecec912360ea16975fe770a547d26b0c0ed258baf6431c75e7,"With fastText at the commit of arXiv v3 and its classification-results.sh, test accuracy with 10 hidden units and bigrams reproduces Table 1 on all eight datasets: 92.2, 96.7, 98.6, 95.7, 63.9, 72.4, 60.2 and 94.6 against 92.5, 96.8, 98.6, 95.7, 63.9, 72.3, 60.2 and 94.6. The unigram row also reproduces, within 0.2 points everywhere. The script own Google Drive download no longer works, so the datasets came from a public mirror of the same tarballs, preprocessed with the script own function. Training speed (Table 2) was not tested."
+2004.01105,Inferring change points in the spread of COVID-19 reveals the effectiveness of interventions,Jonas Dehning; Johannes Zierenberg; F. Paul Spitzner; Michael Wibral; João Pinheiro Neto; Michael Wilczek; Viola Priesemann,2020,Science,10.1126/science.abb9789,Epidemiology,reproduced,Reproduced,4,0,4,7,2026-10-18,https://openrepro.org/papers/2004.01105/,https://openrepro.org/evidence/arxiv-2004-01105-20261018T033025Z.html,https://openrepro.org/badge/2004.01105.svg,90e4d597839451c42440893ea4dea7451ef11eaedc3887da05aa3c80dcc11e78,44,https://openrepro.org/bundles/arxiv-2004-01105-20261018T033025Z.tar.gz,96476d1eefbce44190b9ceef45fa2fa3048bbc9fb9f081efdc17d7b27e297b4a,"Fitting the authors three-change-point SIR model with weekly reporting modulation (repository state on the date of arXiv v3) to German case data until 21 April 2020, using the JHU snapshot bundled with the repository, reproduces all registered values: spreading rates 0.43, 0.25, 0.15 and 0.09 (posterior medians 0.428, 0.251, 0.154, 0.094) and change points around March 6, 15 and 23 (posterior medians March 5.8, 15.2 and 22.6). The credible intervals also match Fig. 3 closely. Only the main three-change-point model was refitted; the 0-2 change-point comparisons and forecasts were not."
+1810.08473,From Louvain to Leiden: guaranteeing well-connected communities,V. A. Traag; L. Waltman; Nees Jan van Eck,2019,Scientific Reports,10.1038/s41598-019-41695-z,Network science,reproduced,Reproduced,4,0,4,6,2026-10-18,https://openrepro.org/papers/1810.08473/,https://openrepro.org/evidence/arxiv-1810-08473-20261018T013946Z.html,https://openrepro.org/badge/1810.08473.svg,578432e9062b72e8e2576db5eeed0bf5ac96e6f5b649204a2e77964f446edc6c,42,https://openrepro.org/bundles/arxiv-1810-08473-20261018T013946Z.tar.gz,86dcbd58c63dcf12282500856962fe469c93d768c950085885ed14083f6a03b4,"Using the authors Java implementation at the release tag matching the paper (networkanalysis 1.0.0) on the SNAP DBLP and Amazon networks, with the paper protocol of 10 random starts of 10 iterations each, all tested Table II values reproduced within 0.0005: DBLP Louvain 0.8267 vs 0.8262 and Leiden 0.8384 vs 0.8387; Amazon Louvain 0.9298 vs 0.9301 and Leiden 0.9340 vs 0.9341. Leiden found higher modularity than Louvain on both networks, as claimed. The other four networks in Table II were out of scope (download blocked, proprietary, or too large for the sandbox), and the badly connected community percentages in Fig. 4 were not tested."
+1812.09970,Synthetic Difference-in-Differences,Dmitry Arkhangelsky; Susan Carleton Athey; David A. Hirshberg; Guido W. Imbens; Stefan Wager,2021,American Economic Review,10.1257/aer.20190159,Economics,reproduced,Reproduced,4,0,4,10,2026-10-18,https://openrepro.org/papers/1812.09970/,https://openrepro.org/evidence/arxiv-1812-09970-20261018T010328Z.html,https://openrepro.org/badge/1812.09970.svg,8b90233f1aba9384bcb8d580e05cbaaf9a5947ebf64d2d62c4f0f4b157f1f7cb,59,https://openrepro.org/bundles/arxiv-1812-09970-20261018T010328Z.tar.gz,722885f65553aeeae11db59b5606e62ccbf078240a65b31dadc29c2ad78a2264,"Running the authors own Table 1 code (vignettes/paper-results.Rmd) at the synthdid commit named in their session-info, with R 4.0.5 and the pinned MCPanel and glmnet commits, reproduced every entry of Table 1 at the reported precision: SDID -15.6 (8.4), SC -19.6 (9.9), DID -27.3 (17.7), MC -20.2 (11.5), DIFP -11.1 (9.5). Point estimates differ from the paper by at most 0.05, and placebo standard errors by at most 0.05. Deviations were limited to platform (x86_64 Linux instead of an M1 Mac) and one non-estimating build dependency. The paper-era synthdid commit is no longer on any branch of the repository but is still retrievable from GitHub by its hash."

ledger.json ADDED Viewed

	@@ -0,0 +1,224 @@

+{
+  "schema_version": 1,
+  "version": "2026-10-19",
+  "generated": "2026-10-19",
+  "site": "https://openrepro.org",
+  "license": "https://creativecommons.org/licenses/by/4.0/",
+  "documentation": "https://openrepro.org/data/",
+  "citation": "https://openrepro.org/CITATION.cff",
+  "feed": "https://openrepro.org/feed.xml",
+  "mcp": "https://mcp.openrepro.org/mcp",
+  "papers_listed": 7,
+  "note": "Listed papers only (papers in author right-of-reply are left out until their publication date); every value comes from the attempt reports.",
+  "papers": [
+    {
+      "arxiv_id": "1708.05031",
+      "title": "Neural Collaborative Filtering",
+      "authors": [
+        "Xiangnan He",
+        "Lizi Liao",
+        "Hanwang Zhang",
+        "Liqiang Nie",
+        "Xia Hu",
+        "Tat-Seng Chua"
+      ],
+      "year": 2017,
+      "venue": null,
+      "doi": null,
+      "field": "Recommender systems",
+      "outcome": "reproduced",
+      "outcome_label": "Reproduced",
+      "headline_held": 4,
+      "headline_held_earlier_code": 0,
+      "headline_total": 4,
+      "claims_checked": 6,
+      "checked": "2026-10-19",
+      "page_url": "https://openrepro.org/papers/1708.05031/",
+      "evidence_url": "https://openrepro.org/evidence/arxiv-1708-05031-20261019T135218Z.html",
+      "badge_url": "https://openrepro.org/badge/1708.05031.svg",
+      "log_head_hash": "04ac3824830007550ce23545b33c04ec2f8c290dd7107cca03355bd8ce4ddfc6",
+      "log_events": 67,
+      "bundle_url": "https://openrepro.org/bundles/arxiv-1708-05031-20261019T135218Z.tar.gz",
+      "bundle_sha256": "3e70c77dc0f0de0d0be0fbe4eeb0b7b66ebb735a00446ace8a237b3426bfcd99",
+      "summary": "The authors Keras code (commit 312aff0, identical in code to the current default branch) was run on MovieLens-1M with the exact README environment (Python 2.7.15, Keras 1.0.7, Theano 0.8.0) on CPU, using the README commands with 8 predictive factors and 20 epochs. All six pre-registered Table 2/3/4 values at 8 factors landed within 0.005 of the paper: NeuMF without pre-training HR@10 0.6848 vs 0.688 and NDCG@10 0.4088 vs 0.410; NeuMF with pre-training (initialised from the authors bundled GMF/MLP weights rather than retrained ones) 0.6798 vs 0.684 and 0.4054 vs 0.403; MLP-3 0.6707 vs 0.671 and 0.3979 vs 0.399. Each value is one run reported at the best-HR epoch, which is the scripts own convention. Mid-attempt the shared cloud droplet was destroyed by another process; the environment and code were rebuilt with the same recipe and only the in-flight MLP run was repeated."
+    },
+    {
+      "arxiv_id": "1606.00931",
+      "title": "DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network",
+      "authors": [
+        "Jared Katzman",
+        "Uri Shaham",
+        "Alexander Cloninger",
+        "Jonathan Bates",
+        "Tingting Jiang",
+        "Yuval Kluger"
+      ],
+      "year": 2018,
+      "venue": "BMC Medical Research Methodology",
+      "doi": "10.1186/s12874-018-0482-1",
+      "field": "Medical statistics",
+      "outcome": "reproduced",
+      "outcome_label": "Reproduced",
+      "headline_held": 4,
+      "headline_held_earlier_code": 0,
+      "headline_total": 4,
+      "claims_checked": 6,
+      "checked": "2026-10-19",
+      "page_url": "https://openrepro.org/papers/1606.00931/",
+      "evidence_url": "https://openrepro.org/evidence/arxiv-1606-00931-20261018T225505Z.html",
+      "badge_url": "https://openrepro.org/badge/1606.00931.svg",
+      "log_head_hash": "b6c148f9a6936f84ea439e56d1c64f0a7dcb49e564e06bcc17ae75d7dd7f225f",
+      "log_events": 63,
+      "bundle_url": "https://openrepro.org/bundles/arxiv-1606-00931-20261018T225505Z.tar.gz",
+      "bundle_sha256": "9b7162f05b98080f2e77db8052c1917f05bd67a5dac3eb3eff03b43872d17cb4",
+      "summary": "Rerunning the first author code at the commit matching arXiv v3 (079dee7, 2017-08-08), with the documented command for each experiment in a reconstructed Python 2.7 / Theano 0.9 / Lasagne environment on CPU, reproduces all six DeepSurv C-index values of Table 1 within 0.01: mean of 10 unseeded runs SUPPORT 0.6186 vs 0.6183, METABRIC 0.6435 vs 0.6434, Rotterdam & GBSG 0.6720 vs 0.6684, WHAS 0.8580 vs 0.8626 (run-to-run sd 0.009), simulated linear 0.7739 vs 0.7740, simulated nonlinear 0.6458 vs 0.6489. The WHAS model file in the repository does not match the WHAS hyper-parameters listed in the paper Table 3, and every run ends with a weight-saving error after the metrics are printed (fixed upstream in December 2017). CPH, RSF and the treatment-recommendation results were not scored."
+    },
+    {
+      "arxiv_id": "1711.10561",
+      "title": "Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations",
+      "authors": [
+        "Maziar Raissi",
+        "Paris G. Perdikaris",
+        "George Em Karniadakis"
+      ],
+      "year": 2017,
+      "venue": null,
+      "doi": null,
+      "field": "Scientific machine learning",
+      "outcome": "reproduced",
+      "outcome_label": "Reproduced",
+      "headline_held": 1,
+      "headline_held_earlier_code": 0,
+      "headline_total": 1,
+      "claims_checked": 2,
+      "checked": "2026-10-19",
+      "page_url": "https://openrepro.org/papers/1711.10561/",
+      "evidence_url": "https://openrepro.org/evidence/arxiv-1711-10561-20261018T234638Z.html",
+      "badge_url": "https://openrepro.org/badge/1711.10561.svg",
+      "log_head_hash": "450dad5c4edbfc2ab0c9dce568dcddd793d19978beba85e4a64f26f6724759da",
+      "log_events": 26,
+      "bundle_url": "https://openrepro.org/bundles/arxiv-1711-10561-20261018T234638Z.tar.gz",
+      "bundle_sha256": "888f02774eeceb719c5891f24233a60b9a06bec62af60dd927f93a94d24d838c",
+      "summary": "Running the authors Burgers.py (the earliest version in their repository, from five months after arXiv v1) with its fixed seeds on CPU gives a relative L2 error of 2.15e-3 for the continuous-time Burgers example, against 6.7e-4 reported. That meets the pre-registered order-of-magnitude headline check (at most 6.7e-3) but not the stricter within-a-factor-of-2 check. The paper own Table 1 shows errors varying between 1e-4 and 1e-2 across neighbouring settings, and this run used a CPU rather than the reported GPU. The Table 1 sweep was not run."
+    },
+    {
+      "arxiv_id": "1607.01759",
+      "title": "Bag of Tricks for Efficient Text Classification",
+      "authors": [
+        "Armand Joulin",
+        "Édouard Grave",
+        "Piotr Bojanowski",
+        "Tomáš Mikolov"
+      ],
+      "year": 2017,
+      "venue": "Conference of the European Chapter of the Association for Computational Linguistics",
+      "doi": "10.18653/v1/e17-2068",
+      "field": "Natural language processing",
+      "outcome": "reproduced",
+      "outcome_label": "Reproduced",
+      "headline_held": 8,
+      "headline_held_earlier_code": 0,
+      "headline_total": 8,
+      "claims_checked": 16,
+      "checked": "2026-10-18",
+      "page_url": "https://openrepro.org/papers/1607.01759/",
+      "evidence_url": "https://openrepro.org/evidence/arxiv-1607-01759-20261018T223713Z.html",
+      "badge_url": "https://openrepro.org/badge/1607.01759.svg",
+      "log_head_hash": "87b5ac6197ec5f210cb74ba6aaa5412fe8e2237bdf845f1afee95be9043de61d",
+      "log_events": 56,
+      "bundle_url": "https://openrepro.org/bundles/arxiv-1607-01759-20261018T223713Z.tar.gz",
+      "bundle_sha256": "2ed5e509997e43ecec912360ea16975fe770a547d26b0c0ed258baf6431c75e7",
+      "summary": "With fastText at the commit of arXiv v3 and its classification-results.sh, test accuracy with 10 hidden units and bigrams reproduces Table 1 on all eight datasets: 92.2, 96.7, 98.6, 95.7, 63.9, 72.4, 60.2 and 94.6 against 92.5, 96.8, 98.6, 95.7, 63.9, 72.3, 60.2 and 94.6. The unigram row also reproduces, within 0.2 points everywhere. The script own Google Drive download no longer works, so the datasets came from a public mirror of the same tarballs, preprocessed with the script own function. Training speed (Table 2) was not tested."
+    },
+    {
+      "arxiv_id": "2004.01105",
+      "title": "Inferring change points in the spread of COVID-19 reveals the effectiveness of interventions",
+      "authors": [
+        "Jonas Dehning",
+        "Johannes Zierenberg",
+        "F. Paul Spitzner",
+        "Michael Wibral",
+        "João Pinheiro Neto",
+        "Michael Wilczek",
+        "Viola Priesemann"
+      ],
+      "year": 2020,
+      "venue": "Science",
+      "doi": "10.1126/science.abb9789",
+      "field": "Epidemiology",
+      "outcome": "reproduced",
+      "outcome_label": "Reproduced",
+      "headline_held": 4,
+      "headline_held_earlier_code": 0,
+      "headline_total": 4,
+      "claims_checked": 7,
+      "checked": "2026-10-18",
+      "page_url": "https://openrepro.org/papers/2004.01105/",
+      "evidence_url": "https://openrepro.org/evidence/arxiv-2004-01105-20261018T033025Z.html",
+      "badge_url": "https://openrepro.org/badge/2004.01105.svg",
+      "log_head_hash": "90e4d597839451c42440893ea4dea7451ef11eaedc3887da05aa3c80dcc11e78",
+      "log_events": 44,
+      "bundle_url": "https://openrepro.org/bundles/arxiv-2004-01105-20261018T033025Z.tar.gz",
+      "bundle_sha256": "96476d1eefbce44190b9ceef45fa2fa3048bbc9fb9f081efdc17d7b27e297b4a",
+      "summary": "Fitting the authors three-change-point SIR model with weekly reporting modulation (repository state on the date of arXiv v3) to German case data until 21 April 2020, using the JHU snapshot bundled with the repository, reproduces all registered values: spreading rates 0.43, 0.25, 0.15 and 0.09 (posterior medians 0.428, 0.251, 0.154, 0.094) and change points around March 6, 15 and 23 (posterior medians March 5.8, 15.2 and 22.6). The credible intervals also match Fig. 3 closely. Only the main three-change-point model was refitted; the 0-2 change-point comparisons and forecasts were not."
+    },
+    {
+      "arxiv_id": "1810.08473",
+      "title": "From Louvain to Leiden: guaranteeing well-connected communities",
+      "authors": [
+        "V. A. Traag",
+        "L. Waltman",
+        "Nees Jan van Eck"
+      ],
+      "year": 2019,
+      "venue": "Scientific Reports",
+      "doi": "10.1038/s41598-019-41695-z",
+      "field": "Network science",
+      "outcome": "reproduced",
+      "outcome_label": "Reproduced",
+      "headline_held": 4,
+      "headline_held_earlier_code": 0,
+      "headline_total": 4,
+      "claims_checked": 6,
+      "checked": "2026-10-18",
+      "page_url": "https://openrepro.org/papers/1810.08473/",
+      "evidence_url": "https://openrepro.org/evidence/arxiv-1810-08473-20261018T013946Z.html",
+      "badge_url": "https://openrepro.org/badge/1810.08473.svg",
+      "log_head_hash": "578432e9062b72e8e2576db5eeed0bf5ac96e6f5b649204a2e77964f446edc6c",
+      "log_events": 42,
+      "bundle_url": "https://openrepro.org/bundles/arxiv-1810-08473-20261018T013946Z.tar.gz",
+      "bundle_sha256": "86dcbd58c63dcf12282500856962fe469c93d768c950085885ed14083f6a03b4",
+      "summary": "Using the authors Java implementation at the release tag matching the paper (networkanalysis 1.0.0) on the SNAP DBLP and Amazon networks, with the paper protocol of 10 random starts of 10 iterations each, all tested Table II values reproduced within 0.0005: DBLP Louvain 0.8267 vs 0.8262 and Leiden 0.8384 vs 0.8387; Amazon Louvain 0.9298 vs 0.9301 and Leiden 0.9340 vs 0.9341. Leiden found higher modularity than Louvain on both networks, as claimed. The other four networks in Table II were out of scope (download blocked, proprietary, or too large for the sandbox), and the badly connected community percentages in Fig. 4 were not tested."
+    },
+    {
+      "arxiv_id": "1812.09970",
+      "title": "Synthetic Difference-in-Differences",
+      "authors": [
+        "Dmitry Arkhangelsky",
+        "Susan Carleton Athey",
+        "David A. Hirshberg",
+        "Guido W. Imbens",
+        "Stefan Wager"
+      ],
+      "year": 2021,
+      "venue": "American Economic Review",
+      "doi": "10.1257/aer.20190159",
+      "field": "Economics",
+      "outcome": "reproduced",
+      "outcome_label": "Reproduced",
+      "headline_held": 4,
+      "headline_held_earlier_code": 0,
+      "headline_total": 4,
+      "claims_checked": 10,
+      "checked": "2026-10-18",
+      "page_url": "https://openrepro.org/papers/1812.09970/",
+      "evidence_url": "https://openrepro.org/evidence/arxiv-1812-09970-20261018T010328Z.html",
+      "badge_url": "https://openrepro.org/badge/1812.09970.svg",
+      "log_head_hash": "8b90233f1aba9384bcb8d580e05cbaaf9a5947ebf64d2d62c4f0f4b157f1f7cb",
+      "log_events": 59,
+      "bundle_url": "https://openrepro.org/bundles/arxiv-1812-09970-20261018T010328Z.tar.gz",
+      "bundle_sha256": "722885f65553aeeae11db59b5606e62ccbf078240a65b31dadc29c2ad78a2264",
+      "summary": "Running the authors own Table 1 code (vignettes/paper-results.Rmd) at the synthdid commit named in their session-info, with R 4.0.5 and the pinned MCPanel and glmnet commits, reproduced every entry of Table 1 at the reported precision: SDID -15.6 (8.4), SC -19.6 (9.9), DID -27.3 (17.7), MC -20.2 (11.5), DIFP -11.1 (9.5). Point estimates differ from the paper by at most 0.05, and placebo standard errors by at most 0.05. Deviations were limited to platform (x86_64 Linux instead of an M1 Mac) and one non-estimating build dependency. The paper-era synthdid commit is no longer on any branch of the repository but is still retrievable from GitHub by its hash."
+    }
+  ]
+}