philipp-fischer's picture lvoegtle's picture
Initial public release
b82fcf1
|
Raw History Blame Contribute Delete
1.94 kB

chartnet_qa_06_difficulty

Download the original Parquet files for the core_permissive configuration of ibm-granite/ChartNet, revision ff6cf062b8bb28c9c29ca315f18b1b890ac39712. Images are embedded in image; save each row's original bytes as <row.id>.png at the release root. Numeric row positions are not image IDs.

Map each JSONL image filename to the source Parquet id column: <id>.png is written from that row's image["bytes"]. The release record's top-level id is a conversation identifier, so use its image filename for the lookup. Run this example from the release root after installing pyarrow and downloading the source shards into /downloads/source/:

from pathlib import Path
import json
import pyarrow.parquet as pq

dataset = "chartnet_qa_06_difficulty"
needed = set()
with (Path(dataset) / (dataset + ".jsonl")).open() as f:
    for line in f:
        for message in json.loads(line)["messages"]:
            for item in message["content"]:
                if isinstance(item, dict) and item.get("type") == "image":
                    needed.add(item["image"])

produced = set()
for path in sorted(Path("/downloads/source").rglob("*.parquet")):
    for batch in pq.ParquetFile(path).iter_batches(columns=["id", "image"]):
        for row in batch.to_pylist():
            filename = str(row["id"]) + ".png"
            if filename in needed:
                data = row["image"]["bytes"]
                if data is None:
                    raise ValueError("Missing embedded image: " + filename)
                Path(filename).write_bytes(data)
            produced.add(filename)

missing = sorted(needed - produced)
if missing:
    raise FileNotFoundError(f"{len(missing)} requested source images were not produced: {missing[:20]}")
print(f"Produced all {len(needed)} requested images")