ShippedLarge

AI diagram editing: validated operation schema, 'Edit Diagram with AI' command (vscode.lm) with preview, applyOperations API

Proposed by Jonathan Miller 1 month agoFunding opened 1 month agoFunded 1 month agoShipped 1 month ago
Acceptance · round 1
Shipped
CINo checks
Automated reviewPass with concerns

This is a thorough, well-structured implementation. The operation module has no `vscode` import, is atomic and is validated, with the JSON Schema generated from the same field table as the validator. The interactive flow is cleanly separated from the VS Code UI, uses a modal preview with Keep positions and Re-layout, and reports selection from the webview. Unit tests are strong and map closely to every acceptance criterion. However, no CI ran and the integration test covering real persistence, undo and save was never executed, and the PR raises the minimum VS Code version to 1.90, so backers should confirm a green CI run before accepting.

Acceptance criteria · 10 of 11 met
  • YESUnit tests cover applyOperations for each implemented operation type, including removing a node with attached connectors and using tempIdsoperations.test.ts has a dedicated test for each of the 9 ops, including 'removeNode also removes attached connectors' and 'lets later operations reference tempIds of new nodes and connectors'.
  • YESA batch with one invalid operation leaves the diagram byte-identical (serialized form equal) and unmutatedThe test 'is atomic: one invalid operation leaves the diagram byte-identical and unmutated' compares serializeDiagram output and a deep snapshot, and a frozen-input test checks the input is never mutated.
  • YESValidator rejects unknown types, missing IDs, duplicate IDs and malformed JSONThe validation tests cover unknown op 'groupNodes', missing ids (including elements removed earlier in the batch), duplicate and existing-id tempIds, invalid JSON strings, non-arrays and malformed fields.
  • YESMocked-provider test verifies the prompt contains the diagram, selected node IDs and schemaaiEdit.test.ts 'sends the diagram, selected node ids and schema to the provider' and the buildEditPrompt test check node ids, the labelled selection section and the full schema JSON.
  • YESWith a mocked provider returning valid operations, the command shows a summary and Cancel leaves the document unchanged'shows a summary and leaves the document unchanged on Cancel' asserts the summary lines, no edits and an identical serialization; it runs `runAIEdit`, not the registered command.
  • YESWith the same mocked provider, Apply updates the document'applies the batch as one edit on Apply, keeping positions' asserts one edit labelled 'Edit with AI' and the renamed node and new connector.
  • YESUnparseable or invalid LLM output shows an error and does not modify the documentA parametrized test covers prose output, a non-array, an unknown op and a missing id, each asserting an error, no preview and an unchanged serialization.
  • YESdiagrammer.applyOperations applies valid operations and returns errors for invalid ones, without any LLM callcommands.ts calls `applyOperations` directly and returns `{summary,idMap}` or `{errors}`; the integration test covers both paths but was not run.
  • YESdocs/operations.md exists and matches the implemented schemadocs/operations.md documents all ops, the id/tempId rules, the schema version, an example, the command call and the groupNodes omission, and unit tests check that every op is listed and that the example batch produces the documented summary.
  • YESNo AI vendor SDK added; provider sits behind DiagramAIProviderpackage.json adds no dependencies, and VsCodeLanguageModelProvider implements DiagramAIProvider using only `vscode.lm`.
  • UNCLEARExisting build, lint and tests passThe builder reports that unit tests, lint, type-check and build pass, but no CI ran and the integration suite was not executed.
Concerns
  • No CI checks ran on this commit, and the builder says the new VS Code integration test (the `applyOperations` round-trip with undo, redo and save) was never run because VS Code could not be downloaded. The persistence and undo path through `applyDocumentEdit` is therefore only exercised by unit-level fakes so far.
  • The minimum VS Code version moves from 1.85 to 1.90 (`engines.vscode`, `@types/vscode`). This is needed for the stable `vscode.lm` API, but it drops support for older VS Code releases for all users, not just AI users.
  • The interactive flow is tested through `runAIEdit` with a fake UI and fake document. The real `vscodeUI` wiring in `commands.ts` (modal buttons, dismiss mapped to cancel, progress cancellation) and the `editWithAI` command handler have no automated coverage.
  • The host stores `message.nodeIds` from the webview whenever it is an array, without checking the elements are strings. This is low risk because `selectedNodeIds` later filters to existing node ids.
  • The builder summary is cut off mid-sentence. The diff itself looks complete and was judged as shown.
CI details
No CI checks ran on this commit.

BackersAccepted
1 accept · 0 rebuild · 0 not voted · quorum 1 of 1
MaintainerMerge

Accepted by the backers and merged by the maintainer.

Ballots · 1
AcceptJonathan Miller

Automated review cost $0.22, counted as builder cost.

Feature: AI Diagram Editing via Structured Operations

Motivation

Users and AI agents should be able to modify Diagrammer diagrams with natural-language instructions. Each instruction is translated into small, validated, structured operations against the native diagram model. The LLM never rewrites the whole diagram JSON.

Scope

1. Operation model (src/ai/operations.ts or equivalent)

  • Define a TypeScript discriminated union and a matching JSON Schema with a schemaVersion field. The operations are:
    • addNode
    • removeNode (also removes attached connectors)
    • renameNode
    • moveNode
    • addConnector
    • removeConnector
    • updateNodeMetadata
    • updateConnectorMetadata
    • applyLayout
  • Operations reference existing elements by ID.
  • New nodes use a caller-supplied tempId. Later operations in the same batch can reference that tempId.
  • Provide a validator that rejects:
    • unknown operation types
    • references to missing IDs
    • duplicate IDs or tempIds
    • connectors to nonexistent nodes
    • malformed fields
    • non-array or unparseable input
  • Provide a pure function applyOperations(diagram, ops) -> { diagram, summary } | { errors }.
    • It is atomic: on any failure, the original diagram is returned unchanged and is not mutated.
    • This core logic must not depend on vscode, so a future MCP server can reuse it.
  • groupNodes is included only if the current native format already supports groups or containers. Otherwise it is omitted, and docs/operations.md says so. Do not change the file format.
  • applyLayout delegates to the existing auto-layout from #14 if it exists. If no layout exists, omit applyLayout and document the omission. Do not build a layout engine.

2. Natural-language command

  • Register Diagrammer: Edit Diagram with AI (diagrammer.editWithAI). It is enabled only when a Diagrammer custom editor is active, via a when clause.
  • The command prompts for an instruction with an input box.
  • The prompt includes:
    • the current diagram (nodes and connectors with IDs, labels and metadata)
    • the selected node IDs and labels, clearly labelled as the selection
    • the operation JSON Schema
    • an instruction to return only a JSON array of operations
  • The default provider uses the VS Code Language Model API (vscode.lm), wrapped behind a DiagramAIProvider interface.
    • Do not add API-key settings, secret storage or custom HTTP client code.
    • If no model is available, show an informative error.
  • The response is parsed and validated. Tolerating surrounding code fences is acceptable. On failure, show an error message and leave the document untouched.

3. Selection awareness

  • The webview posts the current selection (an array of node IDs) to the extension host whenever it changes.
  • Use the existing selection model.
  • If only single selection exists, report that single selection. Do not build a new multi-select UX.

4. Preview and confirm

  • Every AI batch shows a human-readable summary before it is applied, for example:
    • Add 3 nodes
    • Remove 1 connector
    • Rename "API" → "Public API"
    • Connect "Worker" → "Queue"
  • Use a modal message or QuickPick with Apply and Cancel.
  • If auto-layout exists, also offer "Keep positions" or "Re-layout".
  • Cancel or dismiss leaves the document unchanged.

5. Persistence

  • Applied changes go through the existing document edit path, so they persist in the existing file format and undo works.

6. External agent entry point

  • Register the command diagrammer.applyOperations(uri?, ops).
    • If uri is omitted, it uses the active Diagrammer document.
    • It validates and applies the operations with no LLM call and no preview UI.
    • It returns { summary } or { errors }.
  • No MCP server is included.

7. Documentation

  • Add docs/operations.md covering:
    • each operation's schema
    • ID and tempId rules
    • the schema version
    • an example batch
    • how to call diagrammer.applyOperations
    • any omitted operations (groupNodes, applyLayout) and why

Acceptance criteria

  • Unit tests cover applyOperations for each implemented operation type, including removing a node with attached connectors and using tempIds.
  • Unit tests verify that a batch with one invalid operation leaves the diagram byte-identical (serialized form equal) and unmutated.
  • Unit tests verify that the validator rejects unknown types, missing IDs, duplicate IDs and malformed JSON.
  • A unit test with a mocked provider verifies that the prompt contains the diagram, the selected node IDs and the schema.
  • With a mocked provider returning valid operations, the command shows a summary, and Cancel leaves the document unchanged.
  • With the same mocked provider, Apply updates the document.
  • Unparseable or invalid LLM output shows an error and does not modify the document.
  • diagrammer.applyOperations applies valid operations and returns errors for invalid ones, without any LLM call.
  • docs/operations.md exists and matches the implemented schema.
  • No AI vendor SDK is added as a dependency. The provider sits behind DiagramAIProvider.
  • The existing build, lint and tests pass.

Out of scope

  • An MCP server
  • OpenAI-compatible or other providers, API-key settings and secret storage
  • New auto-layout or multi-select UX
  • File format changes, including grouping
  • Repository-wide code modification
  • Voice control
  • Real-time collaboration
  • Applying changes without review in the interactive command

No comments yet.