AgroMind

AgroMind is a benchmark that tests how well large multimodal models understand agricultural remote sensing scenes. It has 28,482 question-answer pairs over 20,850 images taken by UAVs, satellites and ground cameras. The images come from nine public datasets and one private global parcel dataset, and cover 106 regions around the world. By sensor, there are 7,158 UAV, 12,268 satellite and 9,056 camera QA pairs.

The questions cover 4 dimensions and 13 task types, from crop identification and health monitoring to environmental analysis. The authors tested 20 open-source and 4 closed-source models. Models struggled most with spatial reasoning and fine-grained recognition, and several leading models did better than humans.

• Paper: Can Large Multimodal Models Understand Agricultural Scenes? Benchmarking with AgroMind
• Original dataset: AgroMind/AgroMind
• Code: rssysu/AgroMind

Tasks

Dimension	Task	Rows
Spatial Perception	Spatial Localization (SL)	2,036
	Spatial Relationship (SR)	1,286
	Boundary Detection (BD)	1,820
Object Understanding	Object Classification (OC)	3,477
	Pest/Disease Diagnostics (PDD)	2,300
	Growth Status Recognition (GSR)	1,200
Scene Understanding	Scene Comparison (SC)	1,657
	Counting (CO)	4,231
	Area Statistics (AS)	3,176
Scene Reasoning	Visual Prompt Reasoning (VPR)	2,364
	Anomaly Reasoning (AR)	1,521
	Climate Type Reasoning (CTR)	1,073
	Planning (PL)	2,341

Structure

train-0000-of-0001.parquet
images/
  image-train-000-of-NNN.zip
  path_to_shard.parquet

Each row has:

• images: list of {"bytes": null, "path": <sha256>}. The path is the SHA256 of the image bytes and points into a zip shard through images/path_to_shard.parquet.
• id: agromind_<task>_<counter>, for example agromind_pdd_66.
• messages: one user turn (images, then question text) and one assistant turn (answer).
• raw_metadata: JSON string with task, type_id, item_id, level1_id, level2_id, level3_id and raw_image_paths. Some rows also have image_id or isMultiple.

Example:

{
  "id": "agromind_ar_1",
  "messages": [
    {"role": "user", "content": [
      {"type": "image", "text": null},
      {"type": "text", "text": "which anomalous area is in this image of farmland?\nA. cloud_shadow\nB. double_plant\nC. planter_skip\nD. standing_water\nE. waterway\nF. weed_cluster\nG. No anomaly"}
    ]},
    {"role": "assistant", "content": [{"type": "text", "text": "F. weed_cluster"}]}
  ],
  "raw_metadata": "{\"task\": \"AR\", \"type_id\": 9, \"item_id\": 125, \"level1_id\": 4, \"level2_id\": 2, \"level3_id\": 1, \"raw_image_paths\": [\"./Agriculture/images/7268V1Z27_1995-6302-2507-6814.jpg\"]}"
}

Changes from the original data

• The 13 QA files are merged into one train split. The file name is kept as task in raw_metadata.
• Options are added to the question as lines like A. text.
• When the options are images (type_id 4 and 5), the images are given in A to D order and the options read A. Image 1, B. Image 2 and so on.
• Answers combine the letter and the option text, for example B. Citrus. Questions with more than one correct answer list each answer on its own line.
• Number options and bounding box answers are turned into strings. Bounding box values are unchanged. They were already normalized to 0-1 as [xmin, ymin, xmax, ymax].
• raw_metadata keeps the source fields except question, answer, options, image_path, image1 and image2, which are already in messages. All image paths of a row are kept in raw_image_paths.
• Images are stored once, deduplicated by SHA256.

Usage

from agml import loadImageTextToTextDataset

dataset = loadImageTextToTextDataset("Project-AgML/AgroMind")

sample = dataset["train"][0]
sample["images"]    # list of PIL images
sample["messages"]

Citation

@misc{li2025largemultimodalmodelsunderstand,
      title={Can Large Multimodal Models Understand Agricultural Scenes? Benchmarking with AgroMind},
      author={Qingmei Li and Yang Zhang and Zurong Mai and Yuhang Chen and Shuohong Lou and Henglian Huang and Jiarui Zhang and Zhiwei Zhang and Yibin Wen and Weijia Li and Haohuan Fu and Jianxi Huang and Juepeng Zheng},
      year={2025},
      eprint={2505.12207},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2505.12207},
}

This dataset is indexed and structured on https://project-agml.github.io/ as part of Project AgML @ UC Davis. This dataset was reformatted from its original format to match HuggingFace's Imagefolder standards but requires an external module (agml) that processes and returns a HF Dataset object faster than HF module functions.