SynthForensics v1.1
We are working on the next release, which adds videos from two more generators.
- Wan2.2
- MiniMax H3
NeurIPS 2026 · Evaluations & Datasets Track
SynthForensics: Benchmarking and Evaluating People-Centric Synthetic Video Deepfakes
1iCTLab s.r.l. · 2University of Catania · *Equal contribution
State-of-the-art open-source T2V/I2V generators synthesize people increasingly hard to distinguish from authentic footage, while current evaluation suites lag: legacy benchmarks target manipulation-based forgeries, and recent synthetic-video benchmarks prioritize scale over realistic human depiction. We introduce SynthForensics, a people-centric benchmark of 20,445 videos from 8 T2V and 7 I2V open-source generators, paired-source from FF++/DFD reals, two-stage human-validated, in four compression versions with full metadata. In our paired-comparison human study, raters prefer SynthForensics in 71–77% of head-to-head comparisons against each of nine existing synthetic-video benchmarks, while facial-quality metrics fall within the FF++/DFD baseline range. Across 15 detectors and three protocols, face-based methods drop 13–55 AUC points (mean 27) from FF++ to SynthForensics and a further 23 under aggressive compression; fine-tuning closes the gap at a backward cost on legacy benchmarks; training from scratch shows synthetic and manipulation features largely disjoint for most detectors. We release dataset, pipeline, and code.
20,445unique synthetic videos
8 + 7T2V + I2V
generators
81,780video files, 4 compression levels
1.96Munique frames
27.5
hof footage
15detectors benchmarked
Updates
-
Accepted at NeurIPS 2026Evaluations & Datasets Track. The full dataset is on Hugging Face.
-
Extended paper (v2)On arXiv: 20,445 videos from 8 T2V and 7 I2V generators, a human realism study and 15 detectors.
-
First preprint (v1)On arXiv.
Every synthetic video has a real twin.
- 01
Start from a real video
1,363 pristine clips from FaceForensics++ and DeepFake Detection anchor the benchmark and fix its train, val and test splits.
- 02
Describe it in eight fields
VideoLLaMA 3 writes a structured caption. Annotators and an LLM screen it field by field before it is used.
- 03
Generate it with every model
The same caption becomes the prompt for 8 text-to-video generators. Seven image-to-video models also start from a reference frame.
- 04
Same content, different generator
Scene, subject and action stay fixed, so a detector has to learn generation artifacts instead of content.
How the benchmark was built
From 1,363 real clips to 81,780 synthetic files.
Seven steps. Keep scrolling: each one comes at you from the horizon.
Start from real videos
Pristine clips from two established benchmarks. Their official split carries over to every synthetic twin.
1,000 FaceForensics++1,363pristine videos
363 DeepFake DetectionDescribe each one in eight fields
A vision-language model writes a structured caption. Annotators also pick a reference frame for image-to-video.
- Short description
- Dense description
- Background
- Style
- Shot type
- Lighting
- Atmosphere
- Camera movement
Check every caption twice
Human review first, then an LLM screen for sensitive content. Anything flagged is rewritten until both pass.
Caption
Prompt checkby hand, field by field
Automated screeningLlama · 7 categories
- violence
- weapons
- vulnerable individuals
- political figures
- geopolitical references
- national symbols
- identifiable persons
Approved
Generate it with every model
The caption becomes a prompt for 8 text-to-video generators; 7 image-to-video models also animate the reference frame.
Promptfrom the structured caption
Text-to-video 8daVinciSelf-Forcing
Image-to-video 7+ reference framedaVinci
Keep only what looks real
Five annotators review every video. If any of them rejects it, the prompt is edited or the parameters are tuned and the video is generated again, until all five pass it.
Attempt 2
5 annotators
- A1
- A2
- A3
- A4
- A5
- anatomy
- temporal
- rendering
- semantics
- ethics
Edit the prompt, tune the parameters
…the man is seated, speaking with natural mouth movements into a microphone…
seed 1407121513 1407121514guidance 3.0 3.5
Regenerated until all five annotators pass it
Encode four versions
Each accepted video ships raw, lossless and at two compression levels, to test robustness in the wild.
SynthForensics
Every video comes with its prompts, seeds, parameters and environment, ready for training and evaluation.
81,780
synthetic video files
20,445 unique videos, with metadata and official splits Download dataset
20,445 unique videos, with metadata and official splits Download dataset
Eight open-source generators.
Architecturally distinct models from 1.3B to 24B parameters, each pinned to the exact commit used for generation. Seven run in both text-to-video and image-to-video mode.
Wan2.1
14B
T2VI2Vcommit 9737cbaCogVideoX
5B
T2VI2Vcommit 7a1af71SkyReels-V2
14B
T2VI2Vcommit 9351d13MAGI-1
24B · distilled
T2VI2Vcommit 6ff822eSelf-Forcing
Self-Forcing
1.3B
T2Vcommit 33593dfLTX-2.3
22B
T2VI2Vcommit 9e8a28edaVinci-
MagiHuman
MagiHuman
daVinci-MagiHuman
15B · distilled
T2VI2Vcommit f9764ceHelios
14B · distilled
T2VI2Vcommit 5cb7ce9People can't tell either.
A paired-comparison study against nine existing synthetic-video benchmarks, with 118 retained participants and 4,121 ratings per question.
71.5%
of head-to-head comparisons prefer SynthForensics on overall quality
76.5%
prefer it on realism
37.5%
of SynthForensics clips were flagged as fake, against 84.1% for existing benchmarks
Landmark stability · FAN
Landmarks that hold like real faces.
Landmark completeness tracks the real FF++/DFD curves across FAN thresholds, while nine existing synthetic benchmarks fall well below.
Face quality · TOPIQ-NR-Face
Inside the real-video range.
Mean TOPIQ-NR-Face falls within the FF++/DFD real-video range, and SynthForensics is the closest benchmark to FF++ in 1-D Wasserstein distance.
Detectors that excel on FF++ meet a new challenge.
15 face-based and synthetic-video detectors under three protocols. Each row pairs a detector's AUC on legacy data with its AUC on SynthForensics.
AUC on FF++
Zero-shot AUC on SF-FF++
DetectorAUC (%)Gap
* Frame-level detector.
These are the headline numbers. For the full results of every detector, protocol and compression level, read the paper →
Organized for research.
Two subsets, T2V and I2V, each in four compression versions, with per-video metadata, dense captions of the sources and official splits that follow FF++.
SynthForensics/
T2V/
videos/
raw/generator output
canonical/H.264 CRF 0, lossless
crf23/light compression
crf40/heavy compression
metadata/
<generator>/<ID>_<gen>_t2v.json
I2V/
videos/same four versions
i2v_frames/conditioning frames
metadata/7 generators
captions/dense captions of the sources
train.json · val.json · test.jsonFF++ split
LTX-2.3 · I2V · metadata (excerpt)
{
"generator": "LTX-2.3",
"mode": "i2v",
"text_encoder": "gemma-3-12b",
"prompt": "A man in traditional white attire sitting
and speaking, likely in a news studio setting. [...]",
"seed": 1364115978,
"width": 1536, "height": 1024,
"num_frames": 121, "frame_rate": 24,
"num_inference_steps": 30,
"source_caption": "000.json", "source_image": "000.png",
"environment": { "repo_commit": "9e8a28e1…", "gpu_name": "A100 80GB" },
"generation_time_seconds": 264.0
}Get SynthForensics.
The benchmark is hosted on Hugging Face. Accept the terms on the dataset page and access is granted automatically.
Quick start
# accept the terms on the dataset page, then:
hf auth login
hf download SynthForensics/SynthForensics --repo-type dataset --local-dir SynthForensics
# or just the lightly compressed T2V videos
hf download SynthForensics/SynthForensics --repo-type dataset --local-dir SynthForensics \
--include "T2V/videos/crf23/*"Use is limited to non-commercial research and education; cite SynthForensics together with FF++ and DFD, and do not redistribute outside the original repository. Dataset licensed under CC BY-NC 4.0, code under MIT.
The team.
* Equal contribution
Cite.
BibTeX
@article{leotta2026synthforensics,
title = {SynthForensics: Benchmarking and Evaluating People-Centric Synthetic Video Deepfakes},
author = {Leotta, Roberto and Sambataro, Salvatore Alfio and Ragaglia, Claudio Vittorio and
Casu, Mirko and Petralia, Yuri and Guarnera, Francesco and Guarnera, Luca and
Battiato, Sebastiano},
journal = {arXiv preprint arXiv:2602.04939},
year = {2026}
}