A documented test lab

How good are today’s AI agents as CAD engineers?

We measure how well AI agents solve a real engineering task in our environment: read a technical drawing, develop a 3D model from it, and document their own work verifiably. Openly documented method, fixed evaluation standard, own results only — the scoring code itself remains internal.

See the results One case, step by step

0.935 Shape fidelity of the best system, averaged over three runs — the bodies are almost congruent with the reference. claude-opus-5 · 26 cases · best run 0.954

Selected results

Best reconstructions

Six examples from the twelve cases in which claude-opus-5 (run 1) matched the reference shape exactly (shape fidelity 1.00). Clicking a tile loads the interactive 3D view – rotate freely (mouse), zoom (wheel), pan (right mouse button or two fingers).

All 3D models shown are AI reconstructions from the technical drawings — not the original models.

Case 045Claude Code · opus-5 · run 1 · shape fidelity 1.00→ case card
Case 014Claude Code · opus-5 · run 1 · shape fidelity 1.00→ case card
Case 027Claude Code · opus-5 · run 1 · shape fidelity 1.00→ case card
Case 033Claude Code · opus-5 · run 1 · shape fidelity 1.00→ case card
Case 080Claude Code · opus-5 · run 1 · shape fidelity 1.00→ case card
Case 900 · bearing blockClaude Code · opus-5 · run 1 · shape fidelity 1.00 · own demo case→ case card

All 26 cases compared — video next to 3D To the measurements

The sequence

From drawing to evaluation

Overall workflow in five stations: technical drawing, AI agent in an isolated environment, CAD build, delivery, evaluation against a fixed standard
Every measurement passes through the same five stations – from the sheet to the evaluation against a fixed standard.
The agent

AI reads the drawing

A command-line agent (codex or Claude Code) receives a technical drawing and the task of modeling the part parametrically – without access to the solution.

The CAD kernel

FreeCAD as the workbench

All work runs through a REST command interface on a real CAD kernel: primitives, boolean operations, semantic selectors, TechDraw projection and measurement, B-rep validation.

The verification layer

Measurement in five dimensions

The result is measured against a reference model in five dimensions – with an evaluation standard kept unchanged for the entire campaign and ceilings stated per case.

The comparison

All seven systems side by side

Results matrix: seven agent-model systems with dimensional accuracy, feature fidelity, shape fidelity, drawing fidelity and pass rate. Claude Code with opus-5 leads at 77 per cent of cases passed (reachable), ahead of codex with gpt-5.6-sol at 36 per cent.
Means per system across all assessed cases — the same data as the results table.

Summary

The campaign verdict in brief

78/78
Models delivered — the best system (claude-opus-5) skipped no case across three runs.
0.935
Shape fidelity as the mean of three runs: the bodies are almost congruent with the reference (best run: 0.954).
60/78
Cases across three runs met all automatically checkable requirements — on average 20 of 26 per run. Raw-value reference figure: 17/78 — in 13 of the 26 cases the measurement ceiling lies below the fixed threshold.
CAD
What is tested is reconstruction from the drawing — not simulation, manufacturing or tolerances.

In short: the best system already reconstructs parts from technical drawings with remarkable shape fidelity — as a tool with human review, not as a replacement for sign-off by a qualified engineer. CAD skills remain necessary: matching the views, spotting ambiguous dimensioning, checking the result (FAQ). The two pass rates are explained in the results.

Scope

What this test lab is – and what it is not

It is

  • a documented test lab with an openly described method;
  • a measurement of our own results only, in our environment (FreeCAD, REST command interface, rulebook);
  • described precisely enough that metric and environment are fully traceable in substance — the code itself remains internal;
  • reported in both directions – including the agents' divergent interpretations.

It is not

  • not an open leaderboard – third-party submissions are not accepted;
  • no publication of third-party data – no drawings, no reference models, no derived dimension files;
  • not a product or service – this lab measures and reports; there is nothing to buy or commission;
  • no code release – scoring and interface code remain internal, described rather than printed;
  • no ranking claim: single runs scatter, which is why we report ranges and separate subsets.

The idea behind CAD-Agents

An agent that operates a real professional tool, works on the same project as its user, and has to answer for its work against verifiable criteria – that is the core of this approach. For CAD this means: no free-form generation of untraceable one-off solutions, but safe operations at a high level of abstraction, executed on a real geometry kernel, plus a verification layer that judges whether a result is not merely topologically valid but professionally correct.

This lab covers the first stage: drawing → model → self-documentation. Simulation (FEM) and manufacturing (CAM) are later stages and not part of the figures shown here.

To the methodology

See the methodology in detail

Five dimensions, per-case ceilings, the isolation protocol, and the limits of what the figures can say.

Open the methodology