All issues

Search results

Cross-project discussion: benchmark validity, reproducibility and provenance

Status: Open.

#312 In evalplus/evalplus;

· KeilerHirsch opened on Aug 14, 2026

Question: should Mbpp/284's plus tests discriminate == from is independently of string interning?

Status: Open.

#309 In evalplus/evalplus;

· toffee-desuwa opened on Jul 14, 2026

🐛 [TestRemoval] - Mbpp/287 float ground truth from canonical true division mismatches the documented integer task for 17 large-n tests

bugSomething isn't workingSomething isn't working

Status: Open.

#308 In evalplus/evalplus;

· toffee-desuwa opened on Jul 14, 2026

evalplus 0.3.1 on Windows: default config returns ('timeout', []) for every program — a silent false all-clear

Status: Open.

#307 In evalplus/evalplus;

· toffee-desuwa opened on Jul 14, 2026

🤗 [Leaderboard Submission] NemoSmith-4B-Instruct (Nemotron-4B-Instruct + RL Imitation for compilers) HumanEval+ & MBPP+

model eval

Status: Open.

#305 In evalplus/evalplus;

· abhilash1910 opened on Jul 9, 2026

Independent Model Evaluation Request: TemuClaude (OpenAI-compatible API)

Status: Open.

#304 In evalplus/evalplus;

· notSaiful opened on Jul 7, 2026

[REQUEST] - Inthasap-Council-Trinity

model eval

Status: Open.

#302 In evalplus/evalplus;

· hotmakthai opened on Jul 4, 2026

🐛 [TestRemoval] - <Mbpp/102> <Expect camelCase but test list expects PascalCase>

bugSomething isn't workingSomething isn't working

Status: Open.

#301 In evalplus/evalplus;

· Bcwz opened on Jun 29, 2026

[Leaderboard Submission] Gemma 4 31B via Google AI Studio (zero-shot) | HumanEval+ 91.5%

Status: Open.

#300 In evalplus/evalplus;

· hypernab opened on Apr 17, 2026

[Leaderboard Submission] Qwen3.6 35B A3B (Local Q4_K_M) HumanEval+ & MBPP+

Status: Open.

#299 In evalplus/evalplus;

· Iipal opened on Apr 17, 2026

[BUG] some model in Thinking mode, code_extract fail

program contract

Status: Open.

#297 In evalplus/evalplus;

· simplew2011 opened on Apr 14, 2026

asyncio

Status: Open.

#296 In evalplus/evalplus;

· simplew2011 opened on Mar 18, 2026