CONG HOANG “BILL” LE

Can you fine-tune a model that only exists as 2-bit ternary weights? This experiment says yes, without ever keeping an fp16 master copy.

Paste a git diff. The model answers with a short analytical think block, then a lament in the voice of a tragic hero, with the actual remedy spelled out at the end. Either the review opens with "Alas" or it doesn't, which makes the persona an unambiguous instrument for measuring whether ternary training worked.

Model: huggingface.co/korokoboom/Bonsai-2-9b-Shakespeare-SFT · Base: Ternary-Bonsai-2-9B

1. Problem Statement

Standard quantization-aware pipelines keep a higher-precision master copy and quantize on the way into each forward pass. Here there was no stored master. The base stayed as frozen ternary codes for the whole run, and the LoRA delta was trained directly against that grid.

The task: turn a 9B ternary model into a Shakespearean code reviewer, trained on ~9,300 diff + review pairs. Style transfer gives a clean signal to evaluate: the persona is either present or absent.

2. Method

• Everything from the GGUF. Hadamard sign vectors, the tokenizer, and the config are rebuilt from GGUF metadata; the model is decomposed into ternary codes, fp16 per-block scales, and dense sidecar tensors.
• LoRA in rotated weight space. Each of the 200 rotated linear layers gets a rank-64 LoRA pair in f32. The master weight M = ternary + B@A is formed on the fly inside the forward pass.
• STE with hysteresis. The forward pass rounds the master weight onto the ternary grid with a straight-through estimator. A weight must pass 0.60 of a block scale to flip up and fall below 0.40 to flip back, suppressing the flip-flop noise that destabilizes training near rounding boundaries.
• Trit-flip merge. The adapter is discarded; its effect is committed into the GGUF by rewriting the 2-bit codes in place. Same size, format, and inference cost as the original, with no adapter or sidecar weights to merge at serve time.

3. Training

Installing Shakespeare cost 55,603,249 trit flips out of 6.9 billion weights (0.80%). Loss fell from 2.31 to 1.40; gradient norm peaked at 24 early in warmup, then settled around 3.8.

4. Results

• Persona consistency: 15/15 on held-out diffs, with a plain "You are a code reviewer" system prompt and thinking enabled.
• Think quality: analytical and grounded in the actual diff, with no persona leakage into the reasoning.
• Technical substance is the weak axis: the model reliably finds a plausible issue and dramatizes it, lands on the issue a human reviewer flagged about half the time, and occasionally invents problems.

What I learned

• Rank 16 could not hold a persona. Early runs plateaued at loss 2.3 and produced 0/15 persona; rank 64 broke through. Capacity was the binding constraint.
• Hysteresis works. Plain STE runs showed gradient norms climbing to 38 while loss rose; the band stabilizes training and lets loss descend.
• With ~10k examples, epoch 3 is optional. Epoch-boundary loss barely moved, so there was no memorization cliff.
• Watch the token budget. The model thinks longer on hard diffs than its training target; four early eval "failures" were simply the think block hitting the generation cap before the tragedy started.

Limitations

Single task, single persona; technical accuracy is imperfect; it requires PrismML's llama.cpp fork for type-42 ternary tensors and will not load in stock llama.cpp.

A 474M model pretrained from scratch on 10B tokens, then fine-tuned to turn a query and a tool schema into strict JSON.

It is not a chat assistant or a general reasoner: it takes a query and a list of available functions and produces a structured call. The goal is a small model that outputs strictly JSON and could eventually run on a CPU.

Model: huggingface.co/korokoboom/billm-474m-fc · Dataset: korokoboom/billm-fc-synth-v3

1. Results

All scores are out of 60 held-out cases: 50 functions that never appear in the training data, plus 10 negatives where the correct output is a refusal.

Selection is effectively tied. Argument extraction is the gap. Both models have about 440M non-embedding parameters in the same 28-layer transformer body, so the difference is not explained by model size. BiLLM additionally scores 10/10 on negatives and 100% valid JSON.

2. Architecture

Pretrained on 10B tokens from DCLM-Baseline over 19,000 steps, final validation loss 2.85. The base model could produce fluent English but had no instruction following, which is expected at this scale.

3. Three Iterations, Two Silent Bugs

• v1 — the loss bug. The loss compared logits at position t to targets at t instead of t+1. The model learned to copy its input, and loss hit 0.0038. Shifting the targets turned loss into 1.2 and behavior into something trainable, but the output was prose, not JSON: dataset composition decides behavior.
• v2 — the tokenizer bug. The byte-level BPE was not seeded with all 256 byte symbols, so { had no token and encoding silently dropped it. Every target was brace-less, and the model learned to predict brace-less output. Adding { and } and resizing the embedding from 32,000 to 32,001 fixed it. Selection reached 80%, arguments 24%.
• v3 — variation in the data. The v2 model grabbed plausible nouns without understanding which slot they belonged in. v3 forced variation across five axes: query mode, argument position, distractors, schema shape, and type enforcement. Across 20k synthetic examples, selection moved 80% → 96.7% and argument accuracy 24% → 48.3%.

4. What Comes Next

• Diagnose the gap: fine-tune Qwen3-0.6B-Base on the v3 data with the same recipe, and run BiLLM v3 with full fine-tuning instead of LoRA.
• Scale function diversity beyond 200 training functions, with realistic schemas and programmatic label audits.
• Put the loss where the errors are: mask the prompt and upweight argument-value tokens.
• Add schema-constrained decoding (GBNF, Outlines, llguidance) to remove structural errors and isolate the true extraction error rate.
• Grow the eval beyond 60 cases, add a public benchmark such as BFCL, and report confidence intervals.

Honest Scope

BiLLM is currently well behind Qwen3-0.6B: it reliably picks the right function and emits valid JSON, but argument extraction is weak (48.3% full correctness vs 88.3%). The eval is small, training and eval functions are LLM-generated, and performance on real-world tool schemas is untested. If you need a working function-calling model today, use Qwen3-0.6B.

Olin Farm Robotics Challenge, 1st Place, 2024

Perceive, classify, and localize weeds in real time from a downward-facing depth camera, so an autonomous weeder knows exactly where to strike.

1. Problem Statement

Small-scale farms control weeds mechanically, not chemically. But hand-weeding doesn't scale, and blanket herbicide is exactly what these farms are trying to avoid. The Farm Robotics Challenge scenario: a robot drives over crop beds and must remove weeds without touching the crops.

That reduces to a perception problem with three hard constraints:

1. Crop vs. weed discrimination at species level: Both are green plants in dirt. Distinguishing them requires identifying which species each green blob is.
2. Metric localization: A bounding box on an image is useless to a robotic arm. The system must convert pixel detections into real-world coordinates in the arm's frame, through camera intrinsics, depth data, and hand-eye calibration.
3. Real-time on embedded hardware: Detection runs continuously on a NUC while the robot is moving. A 2-second-per-frame classifier means the robot has already driven past the weed before it even has a chance to pick it.

My role on the team covered two stages: segmentation and metric localization. Given a raw RGB-D frame, find every plant, isolate it as a crop-able region, then after classificationconvert those pixel detections into real-world coordinates in the arm's frame. (Species classification was handled by teammates.)

2. Solution Architecture

┌──────────────────────┐
│  Intel RealSense     │  color (1280×720 @30fps)
│  D435 (downward)     │  depth (1280×720 z16, hole-filled)
└─────────┬────────────┘
          │  realsense_frame  (custom RGB+depth+intrinsics msg)
          ▼
┌──────────────────────┐
│  weed_detector node  │
│                      │
│  ① Segmentation      │  HSV green mask → morphology → DBSCAN
│     (pixel space)    │  → bounding boxes + centers + crops
│                      │
│  ② Classification    │  ResNet-18 (Pl@ntNet-300K weights)
│     (species)        │  → species + confidence per crop
│     (teammates' work)│
│                      │
│  ③ Localization      │  deproject pixel + depth → camera frame
│     (metric space)   │  → hand-eye transform → arm frame
└─────────┬────────────┘
          │ weed_info  (Vector3[] in arm frame)
          ▼
   ┌──────────────┐
   │  Weeder arm  │  actuates over detected weeds
   └──────────────┘

Design decisions and why:

• Custom ROS 2 message (RealsenseImage) instead of parallel color/depth topics: color, depth, and camera intrinsics must arrive atomically. A detection's 3D position is only meaningful if the depth pixels and intrinsics are from the same frame. Separate topics guarantee eventual desync. Accepting that minimal frames would be dropped when the camera can not produce both rgb and depth for the same timestamp is a worthwhile trade-off for consistent 3D accuracy.
• Classical CV for segmentation, learned model for classification: Every plant is green; HSV thresholding + morphological open/close is fast, interpretable, and tunable in the field (we carried HSV tuning sliders for lab vs. outdoor lighting). Species-level discrimination is where learning is actually needed. Used a ResNet-18 fine-tuned on Pl@ntNet-300K (1,081 species), matched against the known crops (Lactuca, Carota) with a confidence fallback.
• Depth-based deprojection: Using the RealSense intrinsics (rs2_deproject_pixel_to_point) plus the aligned depth image turns any pixel into a metric 3D point, then a calibrated 4×4 hand-eye transform (camera_to_arm) maps camera frame → arm frame. The calibration constants were measured physically and validated with a ground-truth reference-frame test harness.

Pipeline (one frame):

1. RealSense publisher captures color + depth, hole-fills the depth image, packages with intrinsics, publishes at 10 Hz.
2. weed_detector runs segmentation → N bounding boxes with centers.
3. Each center is deprojected to camera-frame meters, then transformed to the arm frame.
4. Each bounding box is cropped and classified; anything not a known crop is a weed.
5. Weed positions are published on weed_info for the arm; an annotated overlay (labeled boxes: red = weed, tan = crop) is rendered for live debugging.

3. Implementation

Segmentation (my contribution)

The full chain in submodules/plant_id/segmentation.py:

1. HSV green isolation, cv2.inRange over a hue band (outdoor-tuned: H∈[40,100], S≥30, V≥80), then morphological open (3×3, removes speckle noise from soil) and close (10×10, fills holes inside leaves), then hard binarization at the 150 level.
2. Point-cloud compression, the critical performance step. A 1280×720 binary mask produces up to ~921,600 candidate points; clustering that is wasteful because DBSCAN cost scales with point count, not image area. Instead, the binary mask is downscaled to a fixed budget of 10,000 pixels preserving aspect ratio, and only white-pixel Cartesian coordinates are kept. Cluster results are mapped back to full resolution by multiplying by the recorded scale ratio.
3. DBSCAN over plant pixels, eps=3, min_samples=3, n_jobs=-1 on the compressed point cloud. Density-based clustering (rather than connected components) is what merges the leaves of one plant into a single cluster while keeping adjacent plants separate, and noise points (label = -1) are filtered before box extraction.
4. Bounding boxes + centers, per-cluster min/max extents with 10 px padding, mapped back to full-resolution coordinates; centers computed for the deprojection step.
5. Cropping, each bbox is cropped from the original (full-res) frame, so the classifier receives full-quality inputs despite the compressed clustering stage.

The performance work

The original version clustered the full-resolution point cloud. Two changes produced the 6× speedup:

• Resolution budgeting: DBSCAN ran on ~10,000 points instead of the raw binary mask's white-pixel count (typically 10–50× more).
• Single-pass geometry: points are converted to Cartesian once, clustered, scaled once on the way back out, no repeated intermediate copies.

Segmentation latency went from 0.3 s to 0.05 s per frame (6×), fast enough for the pipeline to run at the 10 Hz publish rate with headroom for the classifier.

Testing & tuning harness (testing/)

Because there was no labeled dataset at the start, the repo ships its own evaluation infrastructure:

• IoU / precision / recall harness (iou_helpers.py) comparing predicted boxes against hand-drawn YOLO-format ground truth, with YOLO→VOC box conversion via pybboxes
• Interactive HSV tuner (tune_hsv.py), OpenCV trackbars over a live camera or saved image, for lighting-condition-specific thresholds
• DBSCAN tuning (tune_dbscan.py), visual cluster feedback over parameter sweeps
• End-to-end pipeline runner (run_pipeline.py), same segmentation → classification → localization chain as the ROS node, runnable on saved frames for regression testing
• Reference-frame validation (ref_frame_transform.py), transforms calibrated against a 1-ft-interval reference sheet to quantify hand-eye calibration error

Localization (my contribution)

Pixel → world conversion uses the full RealSense intrinsics model (Brown-Conrady distortion coefficients published with each frame), depth in millimeters, and rs2_deproject_pixel_to_point. The camera-to-arm transform is a composed rigid transform (rotation about x by π for the downward mount, plus a measured translation offset), with the arm-frame output published directly as the actuator's target coordinates.

4. Results

• 1st Place, 2024 Farm Robotics Challenge, the pipeline ran live on the robot at the competition.
• 6× segmentation speedup (0.3 s → 0.05 s per frame) from point-cloud compression, with no loss in detection coverage, this speedup is what let the segmentation run on the robot while it moves.
• The IoU evaluation harness (predictions vs. hand-labeled ground truth) let the team tune HSV/DBSCAN parameters with data instead of eyeballing overlays.

What I took from it

This project is where I learned that robotics perception is about combining a series of classical CV techniques with ML: deprojections, transforms, and thresholds robust enough that the model's output means something in the physical world. The unit that mattered most wasn't accuracy; it was meters.

5. Demo

LLM-driven pick-and-place on a physical 6-DOF arm. My first end-to-end robotics project.

Type "pick up the apple". The robot finds the apple, figures out where it is in 3D, solves for six joint angles, and grabs it.

Repo: github.com/titut/CYPIU · Successor project: ZenNav

1. Problem Statement

Household robots don't get drawers of labeled objects and fixed camera rigs. The question I set for myself: can a cheap hobby arm (myCobot 280, ~$700) + a Raspberry Pi + a webcam do language-directed pick-and-place on unstructured household objects?

That decomposes into four sub-problems, each of which is its own discipline:

1. Language → intent: "Pick up the apple" must become (action: pick up, object: apple), robust to phrasing ("grab that apple", "get me an apple").
2. Intent → pixels: "Apple" means nothing to a robot. The word must ground against an actual detection in the camera frame and YOLO needs to actually detect an apple in the frame.
3. Pixels → world pose: A detection is a 2D box. The arm needs a metric 3D position in its own frame, which means camera calibration, a fixed reference marker, and forward kinematics that match the physical arm.
4. World pose → motor commands: Six joint angles that move the end-effector to the target. The inverse kinematics problem, including the cases where the target is unreachable.

2. Solution Architecture

A ROS 2 package of six cooperating nodes, each owning one stage of the pipeline:

 "pick up the apple"
         │
         ▼
┌───────────────┐   GPT-4.1-nano parses free text into
│   cmd_gui     │   {action, object} via the OpenAI API
└──────┬────────┘   (client node / pipeline orchestrator)
       │  service calls
       ▼
┌───────────────┐   YOLOv4 (ONNX, 416×416, COCO classes)
│ obj_detection │   on the camera stream, resolves "apple"
└──────┬────────┘   to a bounding box
       ▼
┌───────────────┐   AprilTag (36h11, 55mm) detected on the
│apriltag_service│  workspace via tf2_ros + FK-derived arm pose
│ + ik + fk     │   → object pose in the arm's base frame
└──────┬────────┘   → damped least-squares IK → joint angles
       │  /joint_angles (Float32MultiArray)
       ▼
┌───────────────┐   pymycobot serial bridge → real motors
│   movearm     │   publishes /current_angles back (feedback)
└──────┬────────┘
       ▼
┌───────────────┐   gpiozero AngularServo on GPIO 18
│    claw       │   gripper open/close service
└───────────────┘

  + teleop (PS5 DualSense via pydualsense), manual override

Key architectural choices:

• LLM as a parser: GPT-4.1-nano converts commands to structured {action, object} pairs; everything downstream is deterministic robotics code. This keeps the LLM's failure mode bounded (worst case: a mis-parsed noun), rather than letting it emit motor commands.
• Service-based coordination (cypiu_interfaces/Command, std_srvs/SetBool) between orchestrator and perception/actuation nodes, the pipeline is a request/response flow.
• Fixed AprilTag reference frame: A printed AprilTag anchors the world frame, avoiding full SLAM for a static workspace; tf2_ros transforms the tag frame, and the object's position is resolved relative to it. Cheap, robust, and exact enough for tabletop manipulation.

3. Implementation

Implemented IK from scratch

Owning the math means owning the arm.

• Modeling: the arm is described in the product-of-exponentials (screw theory) formalism from Modern Robotics (Lynch & Park): a home configuration matrix M and six screw axes S_list, each measured off the physical arm's geometry.
• Forward kinematics (fk.py) computes the end-effector pose from joint angles, validated against the arm's own reported pose, which is how I caught (and fixed) axis-sign errors that no textbook warns you about.
• Inverse kinematics (ik.py): damped least-squares with joint-limit handling,

dθ = Jᵀ(JJᵀ + λI)⁻¹ · e, λ = 5×10⁻⁴

iterated to 10⁻³ tolerance over up to 200 iterations.

• Line search on the step size: every candidate θ + α·dθ is validated by running forward kinematics on it and checking the resulting pose error norm before acceptance (up to 10 backtracking trials). This keeps the solver from diverging near singularities, a detail that mattered constantly on a 6-DOF arm with a compact workspace.
• Joint-limit clamping: IK solutions are clamped against the myCobot's physical joint ranges (±168° for J1, etc.) before anything reaches the serial port.

Perception

• YOLOv4 → ONNX: weights exported to ONNX and served through onnxruntime (CPU inference on the Pi, no GPU), 416×416 input, COCO classes. obj_detection subscribes to the camera with sensor-data QoS and overlays detections live.
• AprilTag 36h11 via the apriltag_ros wrapper with per-tag geometry in config/tags.yaml; detections arrive as an array with relative transforms.
• Calibration: camera intrinsics in config/camera_info.yaml, plus a hand-written calibration workflow (examples/camera_calibration.py).

Actuation

• movearm queues incoming joint-angle messages and streams them to the myCobot over serial at /dev/ttyAMA0 (1 Mbaud), publishing measured angles back at 10 Hz so the orchestrator sees the arm's actual state.
• The claw is a separate gpiozero servo node with pulse-width-calibrated open/close, deliberately isolated behind a SetBool service so gripper logic can't stall the motion pipeline.
• PS5 (DualSense) teleop via pydualsense for manual recovery and demo control.

4. Results

It works end to end: spoken-or-typed commands → the arm picks up household objects that YOLO can recognize (demo video below).

Honest results, as befits a first build:

• What works: the full pipeline executes reliably for COCO-class objects under consistent lighting; the LLM parsing stage is effectively 100% on well-formed commands; DLS-IK with line search converges across the arm's usable workspace; AprilTag anchoring gives repeatable world-frame poses.
• Where it breaks, and why that's valuable: objects YOLO can't classify (novel household items), IK targets outside the workspace (the solver correctly refuses rather than diverging, the line search sees the error floor), and ambiguous commands ("pick it up" with no referent). Every one of these was mapped deliberately: I probed each boundary condition, documented how the arm behaves when it fails, and built the system so failures stay visible rather than silent.
• The deepest lesson: the hard parts were never the parts I expected. The LLM integration took an afternoon. Making FK match reality, calibrating the camera-to-arm transform, and surviving serial-latency feedback loops took weeks. Robotics is where your code meets physics at the last millimeter.

Why this project exists

Beyond the demo itself, building on ROS 2 end-to-end showed me exactly what the framework provides (message plumbing, TF, launch) and what it costs (codegen ceremony, QoS semantics I didn't control, launch-stack opacity). That diagnosis is the direct origin of ZenNav, where I rebuilt the same ideas (pub/sub autonomy stack, layered safety, session replay tooling) by hand on Zenoh.

5. Demo

One Python host, many agents, each a self-contained folder with a prompt, tools, skills, memory, and a schedule.

A host where an agent is a folder. Drop a directory with a system prompt and some Python functions into toolboxes/, and it's live: chat with it from a terminal UI, an HTTP API, Discord, Telegram, or WhatsApp. It remembers conversations, keeps its own schedule, and can delegate to other agents.

Repo: github.com/titut/agenthost · Companion project: Automation Forge

1. Problem Statement

Agent frameworks in the wild tend to fail in one of two directions: heavyweight platforms that lock your agent's brain behind their API, or notebook-grade scripts that evaporate on restart. I wanted a personal agent infrastructure with a different center of gravity:

1. The agent is a folder: A whole agent, identity, tools, knowledge, schedule, memory, should be inspectable and versionable as plain files and Python functions. No YAML schema that grows until it's a programming language, no external state I can't read.
2. Lifelong memory on a real budget: LLM context windows are lossy and paid for by the token. The agent needs to remember, verbatim recent context, semantic retrieval of the distant past, backed by a boring, reliable SQLite file per agent.
3. Capability without context rot: Real agents accumulate toolsets (research, finance tracking, documents, email, scheduling) far too large to keep in the system prompt at once. The host must let agents swap capability sets at runtime without losing identity or memory.
4. Presence: An agent that only exists when I open a terminal has limited reach. It needs interfaces where I already am (chat platforms), and autonomy (scheduled events, self-managed triggers) independent of any client.

The design bet: most agent frameworks over-abstract. If the unit of an agent is a folder, everything else, hosting, interfaces, memory, scheduling, becomes small, replaceable plumbing.

2. Solution Architecture

                    ┌─────────────────────────────┐
                    │         Agent (core)        │
                    │  OpenAI-compatible LLM loop │
                    └──────────────┬──────────────┘
           ┌───────────────┬───────┴────────┬───────────────────┐
           ▼               ▼                ▼                   ▼
   ┌──────────────┐ ┌─────────────┐ ┌──────────────┐  ┌───────────────┐
   │   Memory     │ │   Toolbox   │ │   Planner    │  │    Events     │
   │ SQLite + RAG │ │  discovery  │ │ DAG multi-   │  │ APScheduler   │
   │ per agent    │ │ (runtime    │ │ step plans,  │  │ (agent-managed│
   │              │ │ switching   │ │ step-by-step │  │  cron, YAML)  │
   └──────────────┘ └─────────────┘ └──────────────┘  └───────────────┘
           ▲               ▲                ▲                   ▲
           └───────────────┴───────┬────────┴───────────────────┘
                                   │ built-in + toolbox tools
                    ┌──────────────┴──────────────┐
                    │        Interfaces           │
                    │ Textual TUI · CLI (--once)  │
                    │ FastAPI + SSE server        │
                    │ Discord · Telegram · WhatsApp│
                    └─────────────────────────────┘

An agent is a directory:

agents/finance/
├── WHOAMI.md      # system prompt, the agent's identity
├── CRITICAL.md    # high-priority instructions injected above everything
├── agent.yaml     # model, memory, built-in tools, secrets, bridging config
├── tools/         # Python functions → LLM tool schemas (auto-discovered)
├── skills/        # Markdown knowledge snippets, loaded on demand via get_skill
└── events.yaml    # scheduled triggers (daily/interval/one-shot, human-readable)

Key design decisions:

• Toolbox switching: Toolsets live in composable toolboxes, directory bundles of tools/ (functions) and skills/ (Markdown). A finance agent swaps to the researcher toolbox mid-conversation to browse the web, then switches back, keeping thread state and memory intact. Toolbox state is per-thread, so two conversations with the same agent can hold different toolsets. Only built-ins persist across switches.
• Skills as lazy-loaded Markdown: Skill summaries sit in the system prompt; full content is fetched via a get_skill built-in only when relevant. Knowledge scales without eating the context window.
• Secrets outside the codebase: API keys live in a KeePass .kdbx database, loaded into environment variables at runtime, nothing sensitive in agent.yaml or git.
• Model-agnostic: The openai SDK pointed at any OpenAI-compatible endpoint (default DeepInfra; Ollama/vLLM work for everything), with reasoning-effort and penalty controls.

3. Implementation

The agent loop (agent.py, ~1,100 lines)

Async OpenAI streaming loop with full tool-calling support: tool schemas are generated from Python functions by signature inspection, executed through a ToolRunner, and results fed back until the model produces a final answer. Context assembly layers, in order: agent identity (WHOAMI.md), critical instructions, skill summaries, active toolbox tools, current datetime grounding, then RAG-recalled memory. Streaming responses propagate to every interface via async generators.

Memory with RAG (memory.py, ~700 lines)

Per-agent SQLite database with three tiers of recollection:

• Recent window, recent messages stored verbatim, replayed into the prompt.
• Semantic retrieval, older conversation content is chunked, embedded, and retrieved by cosine similarity against the current query, so week-old context resurfaces exactly when relevant. Embeddings via bge-m3 through any OpenAI-compatible endpoint.
• Key/value store, for durable facts the agent explicitly wants to keep.

The retrieval pipeline is the same one exposed to the researcher toolbox: chunk → embed → top-k cosine similarity.

Planning (planning.py)

Agents can create structured multi-step plans as a DAG: plan(action="create") builds steps with dependencies and optional per-step toolbox assignment; plan(action="next") executes exactly one step as a focused agent.chat() invocation. Step results are persisted to a task-state database (with statuses: pending/running/completed/failed/skipped/blocked) and injected into later step prompts, so a ten-step task survives context limits, crashes, and thread switches.

Scheduled events (events.yaml + APScheduler)

Schedules are structured YAML (daily, weekday, day-of-week, interval, one-shot; multiple times per trigger; timezone-aware via ZoneInfo), validated by Pydantic. Agents manage their own triggers through the event_tool CRUD tool, and the scheduler hot-reloads when the YAML changes. An agent can decide, mid-conversation, to remind itself to do something tomorrow.

Interfaces

• Textual TUI (chat_tui.py, ~1,400 lines): streaming chat, tool-call/result cards, thread picker, @path file attachment (with transparent text extraction from .docx/.pdf/.xlsx/.csv), and a chatless monitor mode for watching scheduled activity.
• CLI: one-shot queries (--once) for scripting, plain-text loop mode, thread monitoring.
• FastAPI + SSE server: streaming HTTP API for any client.
• Bridges: Discord (mention/prefix triggers, access control, attachments, message edit), Telegram (polling, access control), WhatsApp (Node.js/Baileys bridge with QR pairing).

The toolboxes

Shipped as working examples of the model: finance (18 tools, double-entry ledger, budgets with threshold checks, trend analysis, exchange rates, encrypted backup), researcher (DuckDuckGo search + crawl4ai fetching, chunked and embedded into a local vector store with bge-m3, budgeted query plans), document_writer (DOCX/XLSX/CSV/Markdown generation), gmail, and todo.

4. Results

• In daily personal use as my own assistant host, the finance, research, and document toolboxes run as real daily tools.
• One host, six interfaces, no per-agent glue code: the same folder-based agent is reachable from terminal, HTTP, and three chat platforms. Adding an agent is adding a folder.
• Memory that survives: SQLite + RAG retrieval recovers context far outside the context window, the agent references conversations from weeks ago when they're relevant, at zero marginal token cost for old messages.
• ~8,400 lines of core code doing what multi-thousand-line frameworks do, with every abstraction justified by a concrete use.

What I took from it

The interesting engineering in agents is everything around the model call: what to put in the system prompt, what to retrieve and when, how to bound a tool's blast radius, and how to give an agent agency (schedules, planning, toolbox switching) without giving it fragility. Building the host taught me more about where LLM agents actually fail than any framework tutorial could, and the framework-sized lessons (bounded toolsets, validated schemas, durable memory) are exactly what Automation Forge optimizes next.

5. Demo

Treat the agent, its prompt and tool schemas, as an optimizable artifact, and close the loop with no human inside it.

You have an agent, a system prompt and a set of tool definitions, and a fixed set of test cases with expected outputs. Automation Forge runs the loop for you: execute every case, judge the failures, diagnose why they failed, apply targeted edits to the prompt and tool descriptions, and repeat, until quality thresholds are met or a budget is spent. No human in the loop.

Repo: github.com/titut/automation-forge · Born from: AgentHost

1. Problem Statement

While running live agents on AgentHost, the bottleneck stopped being the model and became the configuration: a system prompt with one ambiguous sentence, or a tool description with one ambiguous parameter, would produce large, inconsistent variance in output quality. The standard fix, hand-editing the prompt, rerunning tests, eyeballing the difference, is slow, unrepeatable, and throws away the reason each edit worked.

The problem statement: given a starting configuration (system prompt + tool schemas) and a fixed set of user-provided test cases, autonomously produce a configuration that passes them reliably.

Three constraints shaped the entire design:

1. No human in the loop: The user provides inputs and the oracle (expected results) up front; everything between is autonomous. Human review happens before and after, not during.
2. The optimizer must not be able to break what it's fixing: The obvious failure mode of prompt-optimization loops is a rewrite-happy editor that improves one case while silently regressing three others. Edits must be structured and permissioned, not free-text.
3. Statistical confidence: LLMs are non-deterministic. A configuration that scores 100% once needs statistical evidence before the loop declares success.

2. Solution Architecture

 User provides (once):                     The optimization loop:
 ┌─────────────────────┐
 │ system_prompt.md    │        ┌──────────────────────────────────────┐
 │ tool_defs.json      │        │  ① Test Harness                      │
 │ cases.json          │───▶    │     run config against every case    │
 │ config.yaml         │        │     → output + reasoning + tool calls│
 └─────────────────────┘        └──────────────┬───────────────────────┘
                                               ▼
                                ┌──────────────────────────────────────┐
                                │  ② Evaluator (LLM)                   │
                                │     pass/fail → failure class from   │
                                │     a fixed taxonomy → diagnosis     │
                                │     → proposed structured edits      │
                                └──────────────┬───────────────────────┘
                                               ▼
                                ┌──────────────────────────────────────┐
                                │  ③ Loop Controller                   │
                                │     aggregate scores · check budgets │
                                │     stop thresholds · pick failures  │
                                └──────────────┬───────────────────────┘
                                               ▼
                                ┌──────────────────────────────────────┐
                                │  ④ Editor (permission-gated)         │
                                │     6 whitelisted operations,        │
                                │     validated → new version          │
                                └──────────────┬───────────────────────┘
                                               ▼
                                ┌──────────────────────────────────────┐
                                │  ⑤ Version Store                     │
                                │     every config + scores persisted, │
                                │     best-so-far always recoverable   │
                                └──────────────────────────────────────┘

Design principles (from SPEC.md, enforced in code):

The six edit operations

The evaluator proposes edits, but can only speak in this grammar, each permission-gated per configuration:

1. edit_system_prompt, replace the prompt or one marked section
2. add_constraint, append a rule
3. edit_tool_description, rewrite a tool's top-level description
4. edit_parameter_description, rewrite one parameter's description
5. rename_tool_display, change display name, never the function binding
6. edit_tool_schema, structural change, disabled by default

Every edit carries a reason string and is validated (Pydantic + permission check) before application. Notably, most gains come from the cheap, safe edits: clarifying a tool description or parameter docstring fixes tool-misuse failures without touching the prompt at all.

3. Implementation

The evaluator's job: statistical confidence

The evaluator LLM receives the full context, current prompt, tool schemas, the case input, the expected-result spec, the agent's actual output, its reasoning trace, its tool calls, and must return strict JSON: pass/fail, a failure class from a fixed taxonomy (missing_step, wrong_format, tool_misuse, hallucination, ignored_constraint, ambiguous_instruction, missing_context, over_complexity), a diagnosis citing specific wording, and ≥ 1 proposed edit for every failing case (enforced, an empty edit list on a failure is invalid output).

Scoring without an LLM oracle (metrics.py)

Deterministic scoring keeps the ground truth trustworthy: exact (normalized text), contains / contains_all (partial credit), json_match (parse + path check), and custom evaluators. The weighted score combines case pass rates with per-case weights, so the loop optimizes what the user actually specified, judged by code.

Regression gating, the part most optimizers skip

This is the feature I'd keep if I kept only one:

• A configuration that reaches the single-run thresholds isn't accepted yet. Each test case is re-sampled samples_per_case times.
• Success requires both the overall pass rate and every individual case's pass rate to clear configured minimums (min_pass_rate, min_case_pass_rate).
• If regression fails, the loop continues optimizing, and the regression results are persisted (version_NNN_regression.json) as evidence.

This directly attacks the non-determinism constraint: the system never accepts a single passing run as proof. The same gate is available standalone via forge regression <run> --samples N, a post-hoc confidence check for any configuration, optimized by Forge or by hand.

Runs are durable artifacts

Every run is a directory with a manifest.json (version history, best-version id, status), per-version case results, and regression records. forge resume continues an interrupted run; forge show-best recovers the winning configuration; budgets mean an unattended overnight run can't bill a surprise. The CLI (forge run / optimize / resume / show-best / regression) is built on click with rich output; every role (agent/evaluator/editor) can point at a different OpenAI-compatible endpoint.

Where it came from

Automation Forge exists because AgentHost made the failure modes visible: watching real agents misuse tools because a description was vague, or ignore constraints buried mid-prompt. The taxonomy in the evaluator is literally the list of failures I watched live agents commit. It optimizes AgentHost-style configurations, which is why tool_defs.json is plain OpenAI tool-schema format and tool implementations are ordinary Python TOOL_HANDLERS.

4. Results

• The loop works end to end, unattended: on the bundled research_task example, an autonomous run took a configuration that failed cases, diagnosed them (tool misuse and format issues), applied targeted edits, and reached the full pass threshold, with the regression gate then re-sampling every case before declaring success. Real run artifacts (manifests, per-case results, regression records) ship in the repo.
• Structured edits demonstrably outperform free-text rewrites in the failure modes that matter: diagnosing tool_misuse and fixing one parameter description is a surgical, reviewable change, something hand prompt-tuning rarely achieves.
• Model-agnostic by construction: the whole loop runs against local endpoints (Ollama/vLLM), making optimization iterations cheap and private.
• Built tested: harness, evaluator, editor permissions, regression gating, and fallback edits all have dedicated test modules.

Honest scope

The current evaluator is single-pass per case, and the oracle modes are exact/match-based rather than rubric-based, "did the agent actually reason well" is out of scope by design (the user owns the oracle). The roadmap is about statistical confidence: more samples, per-class diagnosis depth, and smarter failure-selection heuristics for the next iteration's edits.

What I took from it

Evals are the only way to change an agent without superstition. After building this, I can't unsee it: every prompt tweak is an uncontrolled experiment unless there's a fixed case set, a deterministic judge, and a regression gate. The loop is small; the discipline it enforces is the whole game.

5. Demo