GitChameleon

GitChameleon 2.0 GitChameleon 2.0 is an AI coding benchmark comprising 328 Python-based problems conditioned on specific versions of popular libraries for scientific computing and web development. It evaluates whether AI code generation models can correctly use library APIs as they existed at a particular version — a c

Papers using GitChameleon (3)

GitChameleon: Evaluating AI Code Generation Against Python Library Version Incompatibilities2025GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models2024 · 1 citesGitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models2024