kontext-bench: In-Context Image Editing Benchmark

• Why it matters: Black Forest Labs’ Kontext-Bench is a first-of-its-kind suite of “in-context” image editing tasks. It pairs real images with human-written editing instructions (global edits like style changes and local edits like text removal) to test how well models can follow prompts in the image domain. This specialized benchmark complements recent multimodal models like FLUX.1-Kontext and marks a shift toward evaluating how models apply instructions to images, not just generate them.

  
  

• Key details: The dataset contains 1,026 entries based on 108 unique images, split across categories like Character Reference, Instruction Editing (global/local), Style Reference, and Text Editing. For example, one entry shows a photo of a cat and the prompt “Make the cat look very fat”. This focus on fine-grained edits (rather than open-ended captions) fills a niche in multimodal evaluation.

  
  

• What to watch: Kontext-Bench is brand new (just released with FLUX.1-Kontext-dev), so its usage stats are still ramping up, but it’s already trending among vision/ML safety researchers. Expect it to appear in upcoming papers on visual instruction-following and to influence how future models are trained or fine-tuned for image editing. In a field chasing ever more general vision data, Kontext-Bench is a reminder that task-specific benchmarks can drive progress.

  
  

MegaMath-Web-Pro-Max: Giant Math Reasoning Corpus

• Why it matters: OctoThinker’s MegaMath-Web-Pro-Max is a colossal curated math dataset (over 70 billion tokens) designed to supercharge LLMs on math and reasoning tasks. Originating from the OctoThinker RL-scaling research, this dataset is explicitly tailored for mathematical proficiency. In their new paper, which accompanies the dataset, the authors show that high-quality math corpora like MegaMath dramatically improve both base-model and reinforcement-learning performance on math problems.

  
  

• Key details: MegaMath-Web-Pro-Max was assembled with a multi-step pipeline: sampling raw web math text by year, annotating with GPT-4–style scoring, filtering by quality, and even refining examples with a 70B LLaMA instruct model. It includes tens of millions of math-rich documents spanning proofs, problems, and formal reasoning. The dataset was released under an open (ODC-BY) license alongside the OctoThinker 3B model. This high-end curation (and its 69.2M “rows” of data) positions MegaMath as a go-to resource for anyone training or evaluating math-specialist LLMs.

  
  

• What to watch: Because MegaMath ties directly into a hot topic (scaling RLHF via better data), it’s likely to be picked up by model developers. Keep an eye on upcoming training runs and benchmarks that cite the OctoThinker paper. Also watch whether the community uses MegaMath-Web-Pro-Max as a drop-in dataset for math GPT training or evaluation (e.g. for competitions like GSM8K or new theorem-proving benchmarks).

Visco-Attack: Multimodal Jailbreak Dataset

• Why it matters: Visco-Attack (Visual Contextual Attack) is a chilling new dataset for security researchers. It contains adversarial images paired with harmful prompts designed to jailbreak multimodal LLMs’ safety filters. In practice, it shows how a cleverly crafted visual scene (semantically aligned with a dangerous query) can slip past a model’s guard and coax it into saying something it shouldn’t. As the first public dataset of its kind, Visco-Attack highlights real-world vulnerabilities of image-based LLMs.

  
  

• Key details: The dataset has only 23 examples (a handful of “train” and “validation” cases) but each is a fully worked jailbreak case. For instance, one example includes a visual context of a gang scene plus a text query about “engaging in gang activities”, crafted to override the model’s refusal. Importantly, the authors explicitly warn that the content is offensive/dangerous (the dataset includes a content warning). They even replaced earlier, simpler images (from the prior MM-SafetyBench) with much more semantically precise ones to make the attacks stronger.

  
  

• What to watch: Visco-Attack will be used to evaluate and improve the safety of new vision-language models. It should appear in papers and conferences on AI safety (CVPR/ICLR tracks, for example). Developers of MLLMs (e.g. Google Gemini, OpenAI GPT-4o, Meta LLaVA, etc.) will likely test against it when hardening their filters. The dataset is also a signal: adversarial vision is a growing concern, so expect more such benchmarks.

  
  

  
  

ShareGPT-4o-Image: GPT-4o Vision Prompt Dataset

• Why it matters: FreedomIntelligence’s ShareGPT-4o-Image is a large collection of aligned text prompts and images specifically for OpenAI’s new GPT-4o vision model. Essentially, it’s a distillation of GPT-4o’s knowledge: tens of thousands of high-quality user prompts with corresponding images (and even image-edit prompts). This is the first public dataset tailored for training or evaluating GPT-4o–style multimodal generation, which makes it valuable for any team working on text-to-image or image-to-image LLMs.

  
  

• Key details: The dataset consists of two halves: ~45.7K text-to-image prompts and ~46.5K image-to-image prompts (total ~92K). Prompts range from creative scene descriptions (e.g. “A rebel in comic-book art strokes a dog’s fur…”) to style-transfer or editing tasks. It’s released under Apache-2.0, accompanied by what became the #1 paper of the day last week, and its proofs can be found in their own fine-tuned Janus-4o model. In short, ShareGPT-4o-Image is the GPT-4o training set – coveted by anyone wanting GPT-4o’s capabilities.

  
  

• What to watch: Because many vision-LLMs fine-tuned on GPT-4o outputs will want good training data, this dataset will likely ripple through that community. Look for ShareGPT-4o-Image (or its variants) showing up in model repos and benchmarks. Also watch whether OpenAI or others publish even more GPT-4o data (e.g. as a public sandbox). For now, this set offers a peek at GPT-4o’s “thinking” style.

  
  

Big Picture

This week’s trends underscore a balancing act in the dataset world. On one hand, the “arms race” for bigger, cleaner general corpora continues (as seen by Institutional Books and Essential-Web’s persistent use). On the other hand, researchers are increasingly chasing quality and specialization. Specialized benchmarks — whether for image editing (Kontext), math reasoning (MegaMath), or security attacks (Visco-Attack) — are grabbing attention because they target real challenges in today’s models. In short, size still matters, but cleverness and safety matter too. Expect more hybrid strategies: continued mega-corpora for base knowledge, supplemented by smart, task-focused data to close gaps. The ecosystem is maturing: the next frontier is less about raw scale and more about smart, responsible data construction.

TL;DR: A new wave of niche datasets is upending last week’s winner-take-all race. This week, clever benchmarks and safety sets (Kontext, MegaMath, VisCo-Attack) are trending, even as the old giants (Awesome ChatGPT Prompts, Institutional Books, Essential-Web) keep humming along. The data game is evolving: quality and purpose now share the spotlight with quantity.