|
Download chartnet_qa_06_difficulty/README.md from nvidia/Nemotron-Image-Training-v3.5: direct link, hf CLI and curl.
- Browser
- Download file 1.94 kB
-
https://huggingface.co/datasets/nvidia/Nemotron-Image-Training-v3.5/resolve/main/chartnet_qa_06_difficulty/README.md
- Command line
-
hf download hf://datasets/nvidia/Nemotron-Image-Training-v3.5/chartnet_qa_06_difficulty/README.md
-
curl -L -o README.md https://huggingface.co/datasets/nvidia/Nemotron-Image-Training-v3.5/resolve/main/chartnet_qa_06_difficulty/README.md
1.94 kB
chartnet_qa_06_difficulty
Download the original Parquet files for the core_permissive configuration of ibm-granite/ChartNet, revision ff6cf062b8bb28c9c29ca315f18b1b890ac39712. Images are embedded in image; save each row's original bytes as <row.id>.png at the release root. Numeric row positions are not image IDs.
Map each JSONL image filename to the source Parquet id column: <id>.png is written from that row's image["bytes"]. The release record's top-level id is a conversation identifier, so use its image filename for the lookup. Run this example from the release root after installing pyarrow and downloading the source shards into /downloads/source/:
from pathlib import Path
import json
import pyarrow.parquet as pq
dataset = "chartnet_qa_06_difficulty"
needed = set()
with (Path(dataset) / (dataset + ".jsonl")).open() as f:
for line in f:
for message in json.loads(line)["messages"]:
for item in message["content"]:
if isinstance(item, dict) and item.get("type") == "image":
needed.add(item["image"])
produced = set()
for path in sorted(Path("/downloads/source").rglob("*.parquet")):
for batch in pq.ParquetFile(path).iter_batches(columns=["id", "image"]):
for row in batch.to_pylist():
filename = str(row["id"]) + ".png"
if filename in needed:
data = row["image"]["bytes"]
if data is None:
raise ValueError("Missing embedded image: " + filename)
Path(filename).write_bytes(data)
produced.add(filename)
missing = sorted(needed - produced)
if missing:
raise FileNotFoundError(f"{len(missing)} requested source images were not produced: {missing[:20]}")
print(f"Produced all {len(needed)} requested images")