A practitioner rates a DeepSeek coding agent across six dimensions after a large real-world session, identifying confident misinterpretation of evidence as the critical failure mode.
Adapted from @Whydowecare007I let a DeepSeek-V4.1-Flash coding agent tackle a difficult engineering problem in an existing codebase. The session consumed millions of tokens. There was real value, but also a large gap between persistence and verified progress. My ratings from this session, not a general model benchmark: • Code navigation: 8/10 It traced unfamiliar code and connected relevant components. • Bounded implementation: 7/10 Useful optimizations, narrow bug fixes, and regression tests survived independent review. • Debugging and experimentation: 5/10 It produced detailed experiments, but several did not faithfully reproduce the real execution path. • Judgment and confidence calibration: 4/10 Its biggest weakness was turning "this experiment passed" into "this proves the approach." Small counterexamples overturned some very confident conclusions. • Response to feedback: 8/10 It acknowledged mistakes, withdrew an unsafe direction, restored checks, and retained useful changes. • Token efficiency: 3/10 Millions of tokens bought persistence, but too many rounds revisited measurements without closing the underlying proof obligations. Overall: approximately 6/10 as a supervised coding agent. The most important failure wasn't fabricated code. It was confident interpretation of insufficient evidence. At one point, apparent success improved because previously rejected outputs were allowed through, a change in the definition of success, not a demonstrated solution. How did it compare with the GPT-based coordinating agent? In this session, the coordinator was stronger at challenging assumptions, constructing counterexamples, and separating generated output from verified correctness. But that is not a controlled comparison: reviewing a proposed solution is different from discovering one. The coordinator also benefited from the implementation agent's work, and the reviewer agents themselves made claims that needed correcting. I'd give the coordinator 8/10 for the verification role it performed, not an across-the-board claim that it is "two points better" at coding. The setup that worked best was: bounded implementation, skeptical review, and executable evidence to settle disagreements. My takeaway: DeepSeek-V4.1-Flash was a useful implementer under supervision. I would not trust it, or the reviewing model, to certify its own correctness.
