{"id":8777,"date":"2025-07-20T07:30:59","date_gmt":"2025-07-20T04:30:59","guid":{"rendered":"https:\/\/handoli.com\/index.php\/2025\/07\/20\/data-machina-254\/"},"modified":"2025-07-20T07:30:59","modified_gmt":"2025-07-20T04:30:59","slug":"data-machina-254","status":"publish","type":"post","link":"https:\/\/handoli.com\/index.php\/2025\/07\/20\/data-machina-254\/","title":{"rendered":"Data Machina #254"},"content":{"rendered":"

On the State of AI Coding Agents. <\/strong>\u201cHow could we start using AI to migrate years of messy, flimsy legacy code to a modern stack? <\/em>… Perhaps an AI Code Migration Agent ???<\/em>\u201d <\/p>\n

We\u2019re doing AI chat & espresso at Level 39, One Canada Square. James -a veteran CTO with all the scars- is asking these rather funny, rhetorical questions. There is a deep silence in the room, pensive faces around. Everyone is staring through the massive windows overlooking The City skyline as the sunset strikes. We wonder in perplexity -in the very philosophical and information theory sense- whether AI Coding Agents are fully ready for such tasks in prod, or not and if yes when\u2026 <\/p>\n

Are AI Coding agents any good at solving real-world coding issues autonomously?<\/strong> The team at Princeton Language & Intelligence (PL&I) has come up with SWE-bench, a benchmark for evaluating AI coding agents (paper, code, benchmark)<\/a>. It turns out that current AI Agents are not achieving very good scores in this benchmark yet.<\/p>\n

The PL&I team also open sourced SWE-agent<\/a> an agent that turns LMs (e.g. GPT-4) into software engineering agents that can fix bugs and issues in real GitHub repositories. Checkout the video below with a good hands-on, deep dive.<\/p>\n

\n
<\/div>\n<\/div>\n

Amazon Q<\/strong>. The SWE-bench leaderboard is constantly changing but it seems that Amazon Q Developer Agent<\/a> is for now leading the pack. Amazon Q is a closed model and not very popular in the AI community. It was able to successfully solve only 13.8% out of 2294 tasks. Not a lot really! Here is a vid with a deep dive on Amazon Q.<\/p>\n

\n
<\/div>\n<\/div>\n

Devin the 1st Autonomous AI Engineer? <\/strong> In March, Cognition Labs announced Devin, the world\u2019s first fully autonomous AI software engineer.<\/a> Cognition Labs claimed they were setting a new state of the art on the SWE-bench coding benchmark. Devin went viral but then people in the AI community exposed some tricks used in Devin\u2019s demo. Watch the video below to understand the good, the bad and the ugly of Devin.<\/p>\n

\n
<\/div>\n<\/div>\n

Open source AI community to the rescue: OpenDevin<\/strong>. This started as a small side project and has quickly become one of the most popular AI Coding agents projects. OpenDevin agents<\/a> collaborate with human developers to write code, fix bugs, and ship features. Probably, one of the best open source AI software engineer for developing apps. OpenDevin is now achieving 21% in the swe-bench, the highest score. Checkout the overview below:<\/p>\n

\n
<\/div>\n<\/div>\n

Devika MIT licensed<\/strong>. Devika is an Agentic AI Software Engineer<\/a> that can understand high-level human instructions, break them down into steps, research relevant information, and write code to achieve the given objective. Checkout the vid below.<\/p>\n

\n
<\/div>\n<\/div>\n

Blackbox AI Coding Agents IDE<\/strong>. This is a -still free- pretty amazing AI Coding Agents IDE that comes packed with agents specialised in+30 development languages. The agents can perform natural language to code, chat to code, image to code, plus many s\/w engineering tasks like: bug fixing, unit testing, code translation, API integration, coding docs, coding optimisation\u2026 Apparently millions of devs use it. Checkout Blackbox AI\u2019s playground, agents and features here<\/a>. <\/p>\n

\n
\n
\"\"\n
\n
\n
<\/div>\n
<\/div>\n<\/div>\n<\/div>\n<\/div>\n

<\/p><\/a><\/figure>\n<\/div>\n

GPT-Engineer<\/strong>. Another popular open source AI engineer, gpt-engineer<\/a> lets you: 1) Specify software in natural language, 2) sit back and watch as an AI writes and executes the code, and 3) Ask the AI to implement improvements. Somehow, not sure, but it seems this project perhaps is starting to fall behind other similar projects. In this video Arjan asks: \u201cIs GPT Engineer Actually Useful?<\/em>\u201d<\/p>\n

\n
<\/div>\n<\/div>\n

ChatDev Virtual Software Company<\/strong>. This is an amazing, OSS virtual software company that operates through various intelligent agents holding different roles, including CEP, CPO , CTO, programmer, reviewer , tester, art designer\u2026 These agents form a multi-agent organisational structure and are united by a mission to “revolutionise the digital world through programming.” The agents within ChatDev collaborate by participating in specialised functional seminars, including tasks such as designing, coding, testing, and documenting. Checkout the repo and paper: ChatDev Multi-Agent Collaborative Software Development<\/a>.<\/p>\n

\n
\n
\"AI\n
\n
\n
<\/div>\n
<\/div>\n<\/div>\n<\/div>\n<\/div>\n

<\/p><\/a><\/figure>\n<\/div>\n

Other less ambitious AI Coding Agents for specific s\/w engineering tasks.<\/strong><\/p>\n