MutaFix benchmark
48 injected one-line bugs in 4 open-source Python libraries · Nemotron on Nebius Token Factory · generated 2026-10-25 16:58
40/48 (83.3%)repaired (all tests green)
30/48exact restore of the original line
83.3%ranker top-1 = buggy function
83.3%retriever top-1 (no model)
97.9%bug in the 20 candidates
$0.0081API cost per task
0.0smean ranker call
9.33smean patcher call
By repository
| repository | tasks | repaired | ranker top-1 |
|---|---|---|---|
| toolz | 12 | 9 | 9 |
| more-itertools | 12 | 10 | 12 |
| boltons | 12 | 12 | 7 |
| sortedcontainers | 12 | 9 | 12 |
Every task
| task | bug type | buggy function | ranker pos. | outcome | fix | turns | time | cost |
|---|---|---|---|---|---|---|---|---|
| toolz-001 | NEGCOND | valfilter | 1 | fixed | exact | 2 | 4s | $0.0003 |
| toolz-002 | NEGCOND | check_keywords | 2 | not fixed | - | 8 | 103s | $0.0231 |
| toolz-003 | CMP | curry.__signature__ | 3 | not fixed | - | 8 | 100s | $0.0297 |
| toolz-004 | NEGCOND | get_exclude_keywords | - | not fixed | - | 8 | 145s | $0.0363 |
| toolz-005 | NEGCOND | _has_keywords | 1 | fixed | exact | 4 | 46s | $0.0096 |
| toolz-006 | NEGCOND | countby | 1 | fixed | plausible | 2 | 5s | $0.0003 |
| toolz-007 | NEGCOND | dissoc | 1 | fixed | plausible | 2 | 7s | $0.0005 |
| toolz-008 | NEGCOND | thread_first.<locals>.evalform_front | 1 | fixed | exact | 3 | 22s | $0.0042 |
| toolz-009 | CMP | has_keywords | 1 | fixed | exact | 4 | 46s | $0.0107 |
| toolz-010 | CONST | apply | 1 | fixed | exact | 2 | 6s | $0.0003 |
| toolz-011 | NEGCOND | merge_sorted | 1 | fixed | plausible | 2 | 6s | $0.0007 |
| toolz-012 | NEGCOND | _has_varargs | 1 | fixed | plausible | 2 | 4s | $0.0003 |
| more-itertools-001 | NEGCOND | partition.<locals>.gen | 1 | fixed | exact | 3 | 48s | $0.0041 |
| more-itertools-002 | CMP | one | 1 | fixed | exact | 2 | 37s | $0.0007 |
| more-itertools-003 | BINOP | nth_combination | 1 | not fixed | - | 7 | 136s | $0.0338 |
| more-itertools-004 | BINOP | _sliding_window_deque | 1 | fixed | plausible | 3 | 52s | $0.0042 |
| more-itertools-005 | CONST | set_partitions.<locals>.set_partitions_helper | 1 | fixed | exact | 2 | 36s | $0.0008 |
| more-itertools-006 | NEGCOND | random_derangement | 1 | fixed | exact | 2 | 35s | $0.0007 |
| more-itertools-007 | CMP | numeric_range.count | 1 | fixed | exact | 2 | 35s | $0.0003 |
| more-itertools-008 | CMP | count_cycle | 1 | not fixed | - | 3 | 373s | $0.0057 |
| more-itertools-009 | NEGCOND | chunked.<locals>.ret | 1 | fixed | exact | 1 | 33s | $0.0002 |
| more-itertools-010 | NEGCOND | unique_justseen | 1 | fixed | plausible | 3 | 50s | $0.0031 |
| more-itertools-011 | NEGCOND | classify_unique | 1 | fixed | exact | 2 | 38s | $0.0006 |
| more-itertools-012 | NEGCOND | only | 1 | fixed | exact | 2 | 38s | $0.0007 |
| boltons-001 | NEGCOND | ExceptionInfo.from_exc_info | - | fixed | plausible | 6 | 33s | $0.0085 |
| boltons-002 | CMP | ManyToMany.update | 2 | fixed | plausible | 8 | 111s | $0.0305 |
| boltons-003 | NEGCOND | SpooledStringIO._traverse_codepoints | 5 | fixed | exact | 4 | 43s | $0.0068 |
| boltons-004 | NEGCOND | LRU.__getitem__ | 1 | fixed | exact | 5 | 63s | $0.0138 |
| boltons-005 | NEGCOND | print_exception | 1 | fixed | exact | 3 | 16s | $0.0021 |
| boltons-006 | CONST | Stats._calc_variance | 2 | fixed | plausible | 3 | 21s | $0.0021 |
| boltons-007 | NEGCOND | rotate_file | 1 | fixed | plausible | 2 | 11s | $0.0004 |
| boltons-008 | BINOP | Stats._calc_pearson_type | 1 | fixed | exact | 4 | 57s | $0.0096 |
| boltons-009 | NEGCOND | SpooledIOBase._checkClosed | 1 | fixed | exact | 4 | 29s | $0.0036 |
| boltons-010 | CMP | URL.__ne__ | 1 | fixed | exact | 2 | 10s | $0.0005 |
| boltons-011 | BINOP | BarrelList.iter_slice | 2 | fixed | exact | 3 | 42s | $0.0046 |
| boltons-012 | NEGCOND | find_all_links | 1 | fixed | exact | 6 | 129s | $0.0290 |
| sortedcontainers-001 | NEGCOND | SortedList.__contains__ | 1 | fixed | exact | 2 | 8s | $0.0005 |
| sortedcontainers-002 | CMP | SortedKeyList.count | 1 | fixed | exact | 2 | 9s | $0.0006 |
| sortedcontainers-003 | NEGCOND | SortedKeyList.remove | 1 | fixed | exact | 2 | 9s | $0.0006 |
| sortedcontainers-004 | NEGCOND | SortedSet.discard | 1 | fixed | exact | 3 | 18s | $0.0030 |
| sortedcontainers-005 | NEGCOND | SortedList._delete | 1 | not fixed | - | 8 | 106s | $0.0298 |
| sortedcontainers-006 | CONST | SortedList._pos | 1 | fixed | exact | 3 | 35s | $0.0069 |
| sortedcontainers-007 | NEGCOND | SortedKeyList.index | 1 | fixed | exact | 7 | 139s | $0.0326 |
| sortedcontainers-008 | CONST | SortedList.irange | 1 | not fixed | - | 3 | 35s | $0.0082 |
| sortedcontainers-009 | NEGCOND | SortedKeyList.__contains__ | 1 | fixed | exact | 3 | 20s | $0.0028 |
| sortedcontainers-010 | NEGCOND | SortedList.add | 1 | not fixed | - | 7 | 101s | $0.0212 |
| sortedcontainers-011 | NEGCOND | SortedDict.__ror__ | 1 | fixed | exact | 2 | 9s | $0.0005 |
| sortedcontainers-012 | CMP | SortedSet.count | 1 | fixed | exact | 2 | 8s | $0.0004 |