Operation-Guided Progressive Human-to-AI Text Transformation Benchmark for Multi-Granularity AI-Text Detection
Sondos Mahmoud Bsharat, Jiacheng Liu, Xiaohan Zhao, Tianjun Yao, Xinyi Shang, Yi Tang, Jiacheng Cui, Ahmed Elhagry, Salwa K. Al Khatib, Hao Li, Salman Khan, Zhiqiang Shen
OpAI-Bench is a benchmark dataset that stress-tests AI-text detectors against realistic mixed-authorship documents — not clean human vs. fully AI-generated pairs. Starting from human-written text, it applies five types of AI edit operations across nine progressive revision stages, producing labeled data at document, sentence, and token/span granularity. The key empirical finding is that mixed-authorship intermediate drafts are harder to detect than either endpoint, and that detectability depends on edit type, domain, and revision history — not just the percentage of AI-written content. Detectors tested include 8 document-level, 7 sentence-level, and 2 token/span-level systems.
No production traction yet. The GitHub repo is live at VILA-Lab/OpAI-Bench but has zero citations so far, and the repositories referencing the paper are all paper-aggregation trackers, not implementations building on it. This is primarily useful as a controlled evaluation dataset for teams building or benchmarking AI-text detection systems, particularly those whose pipelines need to handle co-authored or iteratively revised content rather than purely synthetic outputs.
As AI writing assistants become increasingly integrated into real-world drafting and revision workflows, many documents are no longer purely human-written or AI-generated, but instead result from progressive human-AI co-editing. However, existing AI-text detection benchmarks largely focus on final outputs and provide limited understanding of how AI authorship signals emerge, accumulate, or disappear throughout the revision process. We introduce OpAI-Bench, an operation-guided benchmark for studying progressive human-to-AI text transformation across document, sentence, token, and span granularities. Starting from human-written documents, OpAI-Bench constructs nine sequentially revised versions for each sample under predefined AI coverage levels and five representative AI edit operations, covering four domains while preserving complete authorship provenance at multiple granularities. The benchmark supports comprehensive evaluation with 8 document-level detectors, 7 sentence-level detectors, and 2 fine-grained token/span-level detectors. Experiments reveal that AI-text detectability is governed not only by the proportion of AI-edited content, but also by edit operation, domain, and cumulative revision history. Interestingly, we notice that mixed-authorship intermediate versions are often harder to detect than both fully human and heavily AI-edited endpoints, exposing non-monotonic detection patterns missed by existing benchmarks. OpAI-Bench provides a controlled testbed for analyzing whether, when, and how AI-assisted writing becomes detectable under realistic progressive editing scenarios. Our code and benchmark are available at https://github.com/VILA-Lab/OpAI-Bench.