Welcome to Code|AI

Introducing the Code|AI team blog — where we share our journey building AI-powered tools for every developer on the planet.

Building Long-Distance Next Edit Suggestions↗

How we extended GitHub Copilot's Next Edit Suggestions to predict and suggest edits anywhere in a file — not just near the cursor — using a multi-model approach.

Copilot's New Embedding Model in VS Code↗

How we built a new embedding model to improve GitHub Copilot's code understanding in VS Code.

DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models↗

DevBench is a telemetry-driven benchmark designed to evaluate Large Language Models (LLMs) on realistic code completion tasks. It includes 1,800 evaluation instances across six programming languages and six task categories derived from real developer telemetry, such as API usage and code purpose understanding. Unlike prior benchmarks, it emphasizes ecological validity, avoids training data contamination, and enables detailed diagnostics. The evaluation combines functional correctness, similarity-based metrics, and LLM-judge assessments focused on usefulness and contextual relevance. 9 state-of-the-art models were assessed, revealing differences in syntactic precision, semantic reasoning, and practical utility. Our benchmark provides actionable insights to guide model selection and improvement-detail that is often missing from other benchmarks but is essential for both practical deployment and targeted model development.

Pareesa Ameneh Golnari, Adarsh Kumarappan, Wen Wen, Xiaoyu Liu, Gabriel Ryan, Yuting Sun, Shengyu Fu, Elsie Nallipogu