CONG HOANG “BILL” LE

Olin Farm Robotics Challenge — 1st Place, 2024

Perceive, classify, and localize weeds in real time from a downward-facing depth camera, so an autonomous weeder knows exactly where to strike — in world coordinates, on a moving robot.

1. Problem Statement

Small-scale farms control weeds mechanically, not chemically — but hand-weeding doesn't scale, and blanket herbicide is exactly what these farms are trying to avoid. The Farm Robotics Challenge scenario: a robot drives over crop beds and must remove weeds without touching the crops.

That reduces to a perception problem with three hard constraints:

1. Crop vs. weed discrimination at species level. Both are green plants in dirt. Distinguishing them requires identifying which species each green blob is — "plant vs. not-plant" is useless when the crop is lettuce and the weed is a weed.
2. Metric localization, not just pixels. A bounding box on an image is useless to a robotic arm. The system must convert pixel detections into real-world coordinates in the arm's frame, through camera intrinsics, depth data, and hand-eye calibration.
3. Real-time on embedded hardware. Detection runs continuously on a Raspberry Pi-class computer while the robot is moving. A 2-second-per-frame classifier means the robot has already driven past the weed.

My role on the team covered two stages — segmentation and metric localization: given a raw RGB-D frame, find every plant, isolate it as a crop-able region, then convert those pixel detections into real-world coordinates in the arm's frame. (Species classification was handled by teammates.)

2. Solution Architecture

┌──────────────────────┐
│  Intel RealSense     │  color (1280×720 @30fps)
│  D435 (downward)     │  depth (1280×720 z16, hole-filled)
└─────────┬────────────┘
          │  realsense_frame  (custom RGB+depth+intrinsics msg)
          ▼
┌──────────────────────┐
│  weed_detector node  │
│                      │
│  ① Segmentation      │  HSV green mask → morphology → DBSCAN
│     (pixel space)    │  → bounding boxes + centers + crops
│                      │
│  ② Classification    │  ResNet-18 (Pl@ntNet-300K weights)
│     (species)        │  → species + confidence per crop
│     (teammates' work)│
│                      │
│  ③ Localization      │  deproject pixel + depth → camera frame
│     (metric space)   │  → hand-eye transform → arm frame
└─────────┬────────────┘
          │ weed_info  (Vector3[] in arm frame)
          ▼
   ┌──────────────┐
   │  Weeder arm  │  actuates over detected weeds
   └──────────────┘

Design decisions and why:

• Custom ROS 2 message (RealsenseImage) instead of parallel color/depth topics: color, depth, and camera intrinsics must arrive atomically. A detection's 3D position is only meaningful if the depth pixels and intrinsics are from the same frame — separate topics guarantee eventual desync.
• Classical CV for segmentation, learned model for classification. Every plant is green; HSV thresholding + morphological open/close is fast, interpretable, and tunable in the field (we carried HSV tuning sliders for lab vs. outdoor lighting). Species-level discrimination is where learning is actually needed — a ResNet-18 fine-tuned on Pl@ntNet-300K (1,081 species), matched against the known crops (Lactuca, Carota) with a confidence fallback.
• Depth-based deprojection, not homography. Using the RealSense intrinsics (rs2_deproject_pixel_to_point) plus the aligned depth image turns any pixel into a metric 3D point, then a calibrated 4×4 hand-eye transform (camera_to_arm) maps camera frame → arm frame. The calibration constants were measured physically and validated with a ground-truth reference-frame test harness.

Pipeline (one frame):

1. RealSense publisher captures color + depth, hole-fills the depth image, packages with intrinsics, publishes at 10 Hz.
2. weed_detector runs segmentation → N bounding boxes with centers.
3. Each center is deprojected to camera-frame meters, then transformed to the arm frame.
4. Each bounding box is cropped and classified; anything not a known crop is a weed.
5. Weed positions are published on weed_info for the arm; an annotated overlay (labeled boxes: red = weed, tan = crop) is rendered for live debugging.

3. Implementation

Segmentation (my contribution)

The full chain in submodules/plant_id/segmentation.py:

1. HSV green isolation — cv2.inRange over a hue band (outdoor-tuned: H∈[40,100], S≥30, V≥80), then morphological open (3×3, removes speckle noise from soil) and close (10×10, fills holes inside leaves), then hard binarization at the 150 level.
2. Point-cloud compression — the critical performance step. A 1280×720 binary mask produces up to ~921,600 candidate points; clustering that is wasteful because DBSCAN cost scales with point count, not image area. Instead, the binary mask is downscaled to a fixed budget of 10,000 pixels preserving aspect ratio, and only white-pixel Cartesian coordinates are kept. Cluster results are mapped back to full resolution by multiplying by the recorded scale ratio.
3. DBSCAN over plant pixels — eps=3, min_samples=3, n_jobs=-1 on the compressed point cloud. Density-based clustering (rather than connected components) is what merges the leaves of one plant into a single cluster while keeping adjacent plants separate, and noise points (label = -1) are filtered before box extraction.
4. Bounding boxes + centers — per-cluster min/max extents with 10 px padding, mapped back to full-resolution coordinates; centers computed for the deprojection step.
5. Cropping — each bbox is cropped from the original (full-res) frame, so the classifier receives full-quality inputs despite the compressed clustering stage.

The performance work

The naive version clustered the full-resolution point cloud. Two changes produced the 6× speedup:

• Resolution budgeting: DBSCAN ran on ~10,000 points instead of the raw binary mask's white-pixel count (typically 10–50× more).
• Single-pass geometry: points are converted to Cartesian once, clustered, scaled once on the way back out — no repeated intermediate copies.

Segmentation latency went from 0.3 s to 0.05 s per frame (6×) — fast enough that the pipeline runs at the 10 Hz publish rate with headroom for the classifier.

Testing & tuning harness (testing/)

Because there was no labeled dataset at the start, the repo ships its own evaluation infrastructure:

• IoU / precision / recall harness (iou_helpers.py) comparing predicted boxes against hand-drawn YOLO-format ground truth, with YOLO→VOC box conversion via pybboxes
• Interactive HSV tuner (tune_hsv.py) — OpenCV trackbars over a live camera or saved image, for lighting-condition-specific thresholds
• DBSCAN tuning (tune_dbscan.py) — visual cluster feedback over parameter sweeps
• End-to-end pipeline runner (run_pipeline.py) — same segmentation → classification → localization chain as the ROS node, runnable on saved frames for regression testing
• Reference-frame validation (ref_frame_transform.py) — transforms calibrated against a 1-ft-interval reference sheet to quantify hand-eye calibration error

Localization (my contribution)

Pixel → world conversion uses the full RealSense intrinsics model (Brown-Conrady distortion coefficients published with each frame), depth in millimeters, and rs2_deproject_pixel_to_point. The camera-to-arm transform is a composed rigid transform (rotation about x by π for the downward mount, plus a measured translation offset), with the arm-frame output published directly as the actuator's target coordinates.

4. Results

• 1st Place, 2024 Farm Robotics Challenge — the pipeline ran live on the robot at the competition.
• 6× segmentation speedup (0.3 s → 0.05 s per frame) from point-cloud compression, with no loss in detection coverage — this was the difference between "demo that runs offline" and "system that runs on the robot while it moves."
• The IoU evaluation harness (predictions vs. hand-labeled ground truth) let the team tune HSV/DBSCAN parameters with data instead of eyeballing overlays.
• The package was merged into Olin-HAIR-Lab's shared robot codebase and reused by later teams.

What I took from it

This project is where I learned that robotics perception is rarely "train a model" — it's making a chain of deprojections, transforms, and thresholds robust enough that the model's output means something in the physical world. The unit that mattered most wasn't accuracy; it was meters.

LLM-driven pick-and-place on a physical 6-DOF arm — my first end-to-end robotics build

Type "pick up the apple". The robot finds the apple, figures out where it is in 3D, solves for six joint angles, and grabs it.

Repo: github.com/titut/CYPIU · Successor project: ZenNav

1. Problem Statement

Household robots don't get drawers of labeled objects and fixed camera rigs. The question I set for myself: can a cheap hobby arm (myCobot 280, ~$700) + a Raspberry Pi + a webcam do language-directed pick-and-place on unstructured household objects?

That decomposes into four sub-problems, each of which is its own discipline:

1. Language → intent. "Pick up the apple" must become (action: pick up, object: apple) — robust to phrasing ("grab that apple", "get me an apple").
2. Intent → pixels. "Apple" means nothing to a robot. The word must ground against an actual detection in the camera frame — and YOLO needs to actually see an apple, not a concept of one.
3. Pixels → world pose. A detection is a 2D box. The arm needs a metric 3D position in its own frame, which means camera calibration, a fixed reference marker, and forward kinematics that match the physical arm.
4. World pose → motor commands. Six joint angles that move the end-effector to the target — the inverse kinematics problem, including the cases where the target is unreachable.

I deliberately built this alone and on real hardware, because the gaps between "works in a notebook" and "works on a robot" are the actual education.

2. Solution Architecture

A ROS 2 package of six cooperating nodes, each owning one stage of the pipeline:

 "pick up the apple"
         │
         ▼
┌───────────────┐   GPT-4.1-nano parses free text into
│   cmd_gui     │   {action, object} via the OpenAI API
└──────┬────────┘   (client node / pipeline orchestrator)
       │  service calls
       ▼
┌───────────────┐   YOLOv4 (ONNX, 416×416, COCO classes)
│ obj_detection │   on the camera stream — resolves "apple"
└──────┬────────┘   to a bounding box
       ▼
┌───────────────┐   AprilTag (36h11, 55mm) detected on the
│apriltag_service│  workspace via tf2_ros + FK-derived arm pose
│ + ik + fk     │   → object pose in the arm's base frame
└──────┬────────┘   → damped least-squares IK → joint angles
       │  /joint_angles (Float32MultiArray)
       ▼
┌───────────────┐   pymycobot serial bridge → real motors
│   movearm     │   publishes /current_angles back (feedback)
└──────┬────────┘
       ▼
┌───────────────┐   gpiozero AngularServo on GPIO 18
│    claw       │   gripper open/close service
└───────────────┘

  + teleop (PS5 DualSense via pydualsense) — manual override

Key architectural choices:

• LLM as a parser, not a planner. GPT-4.1-nano converts commands to structured {action, object} pairs; everything downstream is deterministic robotics code. This keeps the LLM's failure mode bounded (worst case: a mis-parsed noun), rather than letting it emit motor commands.
• Service-based coordination (cypiu_interfaces/Command, std_srvs/SetBool) between orchestrator and perception/actuation nodes — the pipeline is a request/response flow, not a stream.
• Fixed AprilTag reference frame. Rather than solving full SLAM for a static workspace, a printed AprilTag anchors the world frame; tf2_ros transforms the tag frame, and the object's position is resolved relative to it. Cheap, robust, and exact enough for tabletop manipulation.

3. Implementation

Kinematics from scratch

The core of the project — and where most of the debugging happened:

• Modeling: the arm is described in the product-of-exponentials (screw theory) formalism from Modern Robotics (Lynch & Park): a home configuration matrix M and six screw axes S_list, each measured off the physical arm's geometry.
• Forward kinematics (fk.py) computes the end-effector pose from joint angles — validated against the arm's own reported pose, which is how I caught (and fixed) axis-sign errors that no textbook warns you about.
• Inverse kinematics (ik.py): damped least-squares with joint-limit handling,

dθ = Jᵀ(JJᵀ + λI)⁻¹ · e, λ = 5×10⁻⁴

iterated to 10⁻³ tolerance over up to 200 iterations. - Line search on the step size: every candidate θ + α·dθ is validated by running forward kinematics on it and checking the resulting pose error norm before acceptance (up to 10 backtracking trials). This keeps the solver from diverging near singularities — a detail that mattered constantly on a 6-DOF arm with a compact workspace. - Joint-limit clamping: IK solutions are clamped against the myCobot's physical joint ranges (±168° for J1, etc.) before anything reaches the serial port.

Perception

• YOLOv4 → ONNX: weights exported to ONNX and served through onnxruntime (CPU inference on the Pi — no GPU), 416×416 input, COCO classes. obj_detection subscribes to the camera with sensor-data QoS and overlays detections live.
• AprilTag 36h11 via the apriltag_ros wrapper with per-tag geometry in config/tags.yaml; detections arrive as an array with relative transforms.
• Calibration: camera intrinsics in config/camera_info.yaml, plus a hand-written calibration workflow (examples/camera_calibration.py).

Actuation

• movearm queues incoming joint-angle messages and streams them to the myCobot over serial at /dev/ttyAMA0 (1 Mbaud), publishing measured angles back at 10 Hz so the orchestrator sees the actual arm state, not just the commanded state.
• The claw is a separate gpiozero servo node with pulse-width-calibrated open/close — deliberately isolated behind a SetBool service so gripper logic can't stall the motion pipeline.
• PS5 (DualSense) teleop via pydualsense for manual recovery and demo control.

4. Results

It works end to end: spoken-or-typed commands → the arm picks up household objects that YOLO can recognize. Demo GIF slots below (to be captured from the live rig):

[demo GIF: command "pick up the apple" → detection → reach → grasp]

Honest results, as befits a first build:

• What works: the full pipeline executes reliably for COCO-class objects under consistent lighting; the LLM parsing stage is effectively 100% on well-formed commands; DLS-IK with line search converges across the arm's usable workspace; AprilTag anchoring gives repeatable world-frame poses.
• Where it breaks — and why that's valuable: objects YOLO can't classify (novel household items), IK targets outside the workspace (the solver correctly refuses rather than diverging — the line search sees the error floor), and ambiguous commands ("pick it up" with no referent). Each failure mode is a known failure mode, which is the difference between a demo and an engineering baseline.
• The deepest lesson: the hard parts were never the parts I expected. The LLM integration took an afternoon. Making FK match reality, calibrating the camera-to-arm transform, and surviving serial-latency feedback loops took weeks. Robotics is where your code meets physics at the last millimeter.

Why this project exists

Beyond the demo itself, building on ROS 2 end-to-end showed me exactly what the framework provides (message plumbing, TF, launch) and what it costs (codegen ceremony, QoS semantics I didn't control, launch-stack opacity). That diagnosis is the direct origin of ZenNav — where I rebuilt the same ideas (pub/sub autonomy stack, layered safety, session replay tooling) by hand on Zenoh.

What does a robotics framework actually do for you? This project is the answer, built by hand.

A complete robot autonomy stack — perception, localization, planning, control, safety — running as 8 independent pub/sub nodes with no ROS 2: no DDS graph daemon, no .msg codegen, no launch system. Every piece the framework used to provide, I wrote, versioned, and tested myself.

Repo: github.com/titut/CYR · Origin story: CYPIU

1. Problem Statement

After building CYPIU on ROS 2, I could name what the framework gave me — and what it took. The conveniences were real, but so were the costs: a DDS layer whose QoS semantics I didn't fully control, IDL codegen between every node, a launch system that obscured what was actually running, and a graph daemon whose health determined my system's.

The question this repo answers: what is the minimal, explicit core of a robot autonomy stack — and can one person own all of it?

Concretely, the goals were:

1. Replace the framework, not the functionality: localization, planning, control, and safety as decoupled processes that communicate only through named topics — no shared state, no imports between nodes.
2. Own the transport semantics: message schemas, QoS, staleness policy — visible in my code, not buried in an rmw layer.
3. Hardware-agnostic by construction: the "brains" must run unchanged when the robot hardware changes; drivers are configuration, not code edits.
4. Treat the framework's invisible parts as first-class features: safety latching, message validation, session recording/replay, and tests. These are the parts ROS gives you for free and hand-rolled systems usually forget.

Scope of the current build: a differential-drive robot with LIDAR, IMU, and camera, running in a custom 3D simulator against a polygon world with AprilTags. The sim drivers are the HAL's reference implementation; hardware drivers are a config change away.

2. Solution Architecture

The transport: Zenoh, deliberately configured

Every node is a process that publishes/subscribes JSON on Zenoh keys — zero-config peer discovery, no graph daemon. But the important decisions are the semantics layered on top:

The node graph

                 ┌────────────────────┐
                 │  3D Simulator      │ LIDAR raycast, IMU noise model,
                 │  (world+physics)   │ camera AprilTags, differential drive
                 └─────────┬──────────┘
                           │ sensor/*
     ┌─────────────┬───────┴────────┬──────────────────┐
     ▼             ▼                ▼                  ▼
┌─────────┐  ┌───────────┐  ┌──────────────┐   ┌──────────────┐
│ Apriltag│  │   Pose    │  │  Navigator   │   │  Controller  │
│ Detector│  │ Estimator │  │ A* / RRT*    │   │ pure pursuit │
│         │  │ MCL + ICP │  │ + LLM nav    │   │ + teleop     │
└─────────┘  └─────┬─────┘  └──────┬───────┘   └──────┬───────┘
                   │ estimate/pose │ nav/path         │ cmd/velocity
                   └───────┬───────┴──────────────────┘
                           ▼
                   ┌───────────────┐
                   │  Drive Node   │ motor model + encoder noise
                   │  (+ safety)   │ 3-tier safety zones
                   └───────────────┘

   logger (records ** → JSONL) · sim_viewer · topic_view · map_editor

Eight independent processes. Each knows only topic names — replace any of them (or move them across machines; Zenoh peers discover each other) and nothing upstream notices. start.sh brings up the full stack; there is no other launch machinery.

Layered subsystems

• Perception: simulated LIDAR (raycast against polygon world, per-ray footprint-aware clearance), IMU with a noise model, camera-based AprilTag detections at ~30 Hz, structured through the detector node.
• Localization: Monte Carlo Localization — a particle filter fusing LIDAR (beam model), AprilTags (absolute-pose Gaussian likelihood with camera-noise covariance), and a complementary heading filter that jointly estimates gyro bias and wheel-radius scale; Gauss-Newton ICP scan matching with coordinate-descent fallback, gated off while tags are actively correcting; automatic re-seeding when a measurement contradicts every hypothesis.
• Planning: footprint-aware A (8-connected, corner-cut prevention, octile heuristic, greedy line-of-sight smoothing) with RRT behind the same planner interface as a drop-in alternative.
• Control: regulated pure pursuit — curvature from a lookahead point, speed limited by centripetal acceleration and goal distance, rotate-in-place at sharp corners, bang-bang recovery when far off-path, teleop with hard priority.
• Safety: three tiers keyed on minimum clearance — slow-down (< 1.0 m, distance-scaled speed and turn-rate caps), stop (< 0.15 m), and a latched e-stop (< 0.05 m) that never auto-clears, requires explicit reset, and permits only slow reverse so the robot can back out without instantly re-latching. Modeled on IEC 61508 / ISO 3691-4 concepts: on hardware, the stop tier is the safety function on a rated channel, separate from best-effort navigation.
• HAL: DriveDriver, LidarDriver, ImuDriver, CameraDriver interfaces. Brains read/write topics only; new hardware = implement an interface, register it, select it in robot.yaml.

Tooling as first-class citizens

• Recorder/replayer: the logger subscribes to ** and writes every topic to JSONL with wall + monotonic timestamps; the replayer republishes with --speed pacing and topic filters. Deterministic replay (seeded nodes + --speed 0) turns "weird behavior in the field" into a reproducible unit test. This is my rosbag, and I understand it completely.
• Observability: topic_view (live topic inspector), sim_viewer (visualization), map_editor (visual map authoring).
• Tests: ~2,700 lines across 24 modules — kinematics, message schema, particle filter, ICP, A*, RRT, safety latching, replay — runnable via make test / make coverage.

3. Implementation

The schema registry (core/messages.py)

The piece I'd defend hardest. Every topic's payload has a validator; encode() stamps a wire-format version and validates before sending; decode() checks version and every field's type on receipt. Failures raise topic-qualified SchemaErrors that nodes log — bad messages are loud, never silently dropped. The registry is one dict: adding a topic gives every producer and consumer identical type checks. It's .msg IDL's guarantees without the codegen.

The pose estimator

The most intricate node (~850 lines), and the one where the QoS design pays off:

• LIDAR arrives on a RingChannel(1) — the filter processes exactly one scan per loop iteration, always the newest, never a backlog.
• AprilTag detections are consumed on a dedicated thread so they fuse at camera rate rather than LIDAR rate.
• The beam model weights particles; fresh tag detections re-weight the cloud by Gaussian likelihood derived from the camera noise model; repeated detections converge the cloud onto the tag mean; a measurement contradicting every particle triggers recovery re-seeding inside the fusion step.
• ICP refinement runs only when the scan is structured enough to localize against and no tag has fused recently — so the two absolute-referencing mechanisms never fight.

The drive node and safety

The drive node models the hardware: unicycle → differential wheel speeds, torque-limited first-order velocity loop, encoder noise. The safety response is computed from minimum body-to-surface clearance (each LIDAR ray minus the footprint at that angle), with the three tiers described above. The design comment in the code is explicit about the hardware boundary: which tier is the safety function, which channel it belongs on (IEC 61508 / ISO 3691-4), and why the latch must be manually re-armed.

Navigation + LLM

nav/goal targets come from map clicks or natural language: the room layout is sent to an LLM (DeepSeek via the OpenAI-compatible API — equally happy with Ollama or vLLM locally) with the command, and it resolves "go to the kitchen near the corner" into target coordinates. The LLM proposes; A* and the safety tiers dispose — the language model never touches anything but a goal point.

What I rebuilt by hand, on purpose

launch → start.sh (process supervision, stale-node cleanup, signal trapping). rosbag2 → logger + replayer. .msg IDL → schema registry. QoS profiles → explicit per-topic choices. node diagnostics → topic_view. Each replacement is smaller than the thing it replaced and I can explain every line of it.

4. Results

• A complete, tested autonomy stack with no ROS dependency — 8 nodes, ~2,700 lines of tests, one pip install zenoh and a shell script instead of a workspace build system.
• Transport semantics I can defend line by line: I can explain exactly what happens to a LIDAR sample under congestion, why the estimator can never see a stale scan, and what a malformed message does on every topic — because all three behaviors are in my code, not an rmw implementation detail.
• Deterministic debugging: record a field session, replay at any speed with any topic filter, seed the nodes, and reproduce the exact run. The failure modes that motivated "just re-run it and hope" are gone.
• The HAL thesis is standing up so far: the localization and planning stacks have never needed a change for the simulators they run against; drivers are YAML.

Where it's going

docs/ROADMAP.md is a prioritized backlog (P0–P3, sized S–XL): real camera model + AprilTag detection, SLAM with scan-to-submap and loop closure, dynamic obstacle tracking, costmap layering, recovery behaviors — the gap between "well-structured 2D localization demo" and "universal Robotics OS," written down as tickets.

What I took from it

The honest version: I now know why ROS 2 is the way it is — and I can also say precisely which 20% of it a small robot needs. Building the framework instead of using it changed how I read every robotics system: QoS profiles, schema versioning, safety latching, and replay tooling are no longer features of a framework; they're decisions, and I've now made all of them once, by hand.

Fine-tuning a 7B Vision-Language-Action model on up to 1,000 human demos of a task robots are famously bad at: precise insertion.

Planned project — writeup describes the design, not results. Take a commodity cobot arm, fine-tune OpenVLA 7B on up to 1,000 human demonstrations of plugging a charger into a phone (RGB-D, RealSense), and get an end-to-end policy that goes from camera pixels to a successful charge — no hand-written perception, no hand-tuned IK staging.

Honest expectations up front: I expect this to mostly fail. Contact-rich insertion without force feedback is at the research frontier; a 7B generalist policy fine-tuned by one person on a hobby arm has every reason to score near zero. I'm doing it anyway — the value is in the measured gap between human-trivial and robot-possible, the dataset itself, and the scaling curve, not the success rate. Status: data collection rig in design. This page will be rewritten with real numbers as phases complete.

1. Problem Statement

Everything I've built so far — CYPIU, ZenNav — shares the same skeleton: a classical perception stack feeds a classical planner. It works, but each stage (detector, pose estimator, planner) is a hand-engineered component with hand-tuned handoffs. VLA models promise to collapse that pipeline into one policy: camera frames + language instruction → robot actions, learned end-to-end from demonstration.

I want to test that promise on a task that's deliberately hostile to the classical pipeline:

Charging a phone — insert a charging cable's plug into the phone's port.

Why this task specifically:

• Contact-rich and precision-critical. The plug must enter a slot with millimeter-scale tolerance, through contact forces that a pure position controller will fight. Insertion tasks are the canonical hard case for visuomotor policies.
• Deceptively simple for humans. A person does it in two seconds while barely looking. The gap between "trivially demonstrated" and "trivially learned" is exactly what I want to measure.
• A real, closed-loop success criterion. The phone either charges or it doesn't. No proxy metrics — the task grades itself.
• Bounded dataset size. 1,000 demos is large for a solo project, small by foundation-model standards — which makes the interesting question concrete: how far does fine-tuning a pretrained 7B VLA get you with one camera and one task?

2. Solution Architecture (planned)

┌────────────────────────────────────────────────────────────┐
│                    DATA COLLECTION                          │
│  human demonstrator + cobot arm (kinesthetic guidance)     │
│  RealSense RGB-D @ fixed wrist/scene mount                 │
│  + arm proprioception logged in lockstep with frames       │
│      → (image, language, action) triples → RLDS dataset    │
└─────────────────────────┬───────────────────────────────────┘
                          ▼
┌─────────────────────────────────────────────────────────────┐
│              FINE-TUNING  (OpenVLA 7B)                      │
│  LoRA / QLoRA on the OpenVLA checkpoint                     │
│  input: 224×224 RGB + "plug the charger into the phone"     │
│  output: discretized action tokens (ΔEE pose + gripper)     │
└─────────────────────────┬───────────────────────────────────┘
                          ▼
┌─────────────────────────────────────────────────────────────┐
│              DEPLOYMENT LOOP                                │
│  RealSense frame → policy → action chunk → cobot arm        │
│  → next frame … (closed loop until insertion or timeout)    │
└─────────────────────────────────────────────────────────────┘

Planned stack:

• Policy: OpenVLA 7B — open-weights VLA (Llama-2 backbone + fused DINOv2/SigLIP vision encoders), pre-trained on 970k Open X-Embodiment episodes across hundreds of robots. Fine-tuned, not trained: the value is starting from generalist robot knowledge.
• Robot: a cobot arm with position-controllable joints and gripper (myCobot 280 class — already characterized in CYPIU). Low payload is acceptable: a phone + cable is ~200 g.
• Camera: RealSense RGB-D, a fixed third-person mount covering the workspace — chosen over a wrist cam for v1 to keep the visual distribution consistent across 1,000 demos.
• Action space: OpenVLA's discrete action tokens (256 bins per dimension) over end-effector deltas + gripper open/close, at a fixed control frequency.
• Fine-tuning: LoRA/QLoRA on a single 24 GB GPU (quantized base weights), batched over the RLDS-converted dataset.

The critical data decision — and the honest risk. Human demonstrations filmed with only a camera don't come with action labels; OpenVLA learns (image, instruction) → action, so every frame needs the robot's action too. The plan is kinesthetic teaching: a human physically guides the cobot through the insertion while the arm logs its own joint/EE state, so proprioception is the action label — recorded in lockstep with the RealSense frames. Filming a human hand doing it (no arm labels) is the tempting shortcut and the known trap; it's explicitly out of scope for v1.

3. Implementation Plan

Phase 0 — Rig (week 1–2) - Mount arm + RealSense; fix workspace geometry; lock camera exposure/white balance. - Synchronized logger: RealSense frames + arm proprioception + gripper state → timestamped episodes, RLDS-ready from day one. - Pilot 50 demos to find the failure modes of the data collection itself (workspace occlusions, reset procedure consistency, demographic variance in how people guide the arm).

Phase 1 — Dataset (weeks 2–6) - 1,000 demos across ≥ 3 demonstrators, varying: phone position (a grid over the workspace), phone orientation, cable approach direction, lighting. Every episode: reset → language command → guided insertion → success/failure label. - Deliberately log failures too — a small fraction of failure episodes may be more valuable to the policy than a perfect set, and success/failure metadata enables filtering experiments later. - Dataset hygiene: dedupe, verify proprioception-frame sync (drift is the silent killer of VLA fine-tunes), train/val split by demonstrator not by episode (so validation measures generalization to new humans).

Phase 2 — Fine-tuning (weeks 4–7, overlapping) - Baseline first: zero-shot OpenVLA on the task, recorded as the number to beat. (Expectation: near-0% — the base model has never seen this plug; documenting that matters.) - QLoRA fine-tune on the collected episodes; hold out a validation demonstrator; track validation success rate, not training loss. - Ablations as compute allows: episode-count scaling (the sample-efficiency curve is arguably the most publishable artifact of the project).

Phase 3 — Evaluation & deployment - Fixed protocol: N rollouts per condition (phone at grid positions × orientations × lighting), success = charge detected at the port. - Failure taxonomy on rollouts: approach error, insertion slip, premature gripper release, oscillation at contact — the same instinct as Automation Forge's failure taxonomy, applied to a physical policy. - Closed-loop vs. open-loop comparison (re-plan every step vs. execute a predicted chunk) — insertion tasks are where the difference should show.

The milestone ladder — where the point of no return is

The full 1,000-demo run is the stretch goal, not the commitment. The project is structured so each milestone is informative on its own:

1. Milestone 1 (committed, ~3–4 weekends): rig + 50–100 demos + zero-shot baseline + one small QLoRA fine-tune. This answers the only question that matters: does fine-tuning on my data move the success rate at all? Everything after this is optional and nothing before it is wasted — a "yes" justifies scaling; a "no" is the most interesting negative result a portfolio can hold.
2. Milestone 2 (stretch): scale to 1,000 demos, produce the data-scaling curve.
3. Milestone 3 (stretch): closed-loop comparisons and out-of-grid generalization.

4. Expected Results & Success Criteria

Expected outcome: poor — stated plainly, on purpose. The realistic prediction is that the fine-tuned policy performs meaningfully above the zero-shot baseline but well below reliability — low single-digit to low double-digit success percentage on the hardest contact phase, with most rollouts failing at the insertion. If I'm wrong in either direction, the result is worth publishing on this page; if I'm right, the artifact is the measured gap, the dataset, and the failure taxonomy.

Targets (to be replaced by measured numbers):

• Primary (Milestone 1): fine-tuned policy measurably above the ~0% zero-shot baseline — any statistically real lift from 50–100 demos validates the pipeline.
• Secondary (stretch): data-scaling curve (50/300/1000 episodes); generalization to a phone pose outside the training grid; one demonstrator held out entirely. The 60%-success bar for the full run is an aspiration, not a promise.
• Deliverables (guaranteed regardless of policy performance): public RLDS dataset (RGB-D + proprioception episodes), fine-tuned checkpoints, evaluation harness, failure taxonomy — and a writeup of whatever the numbers turn out to be.

Known risks, ranked:

1. Contact dynamics dominate. VLAs see pixels, not forces; insertion near the slot may stall. Mitigation if needed: visual servoing assist around the final centimeter, or an arm with force sensing.
2. 7B inference latency on available hardware — if the loop runs at 2 Hz, the policy must rely on slower, more deliberate motions (which kinesthetic data can encode).
3. Demo distribution collapse — 1,000 near-identical trajectories teach memorization, not robustness. The diversity protocol in Phase 1 exists for this.
4. Label noise from kinesthetic teaching — human guidance is shaky; smoothing/filtering pass planned before training.

Why this project

Every project so far built a piece of the classical autonomy stack by hand — this one tests the thesis that's currently reshaping the field: that a sufficiently general model, fine-tuned on enough task data, replaces the pipeline. Whether it works or fails measurably, the result is the most interesting thing I'll have built.

One Python host, many agents — each a self-contained folder with a prompt, tools, skills, memory, and a schedule.

Not a framework you configure — a host where an agent is a folder. Drop a directory with a system prompt and some Python functions into toolboxes/, and it's live: chat with it from a terminal UI, an HTTP API, Discord, Telegram, or WhatsApp. It remembers conversations, keeps its own schedule, and can delegate to other agents.

Repo: github.com/titut/agenthost · Companion project: Automation Forge

1. Problem Statement

Agent frameworks in the wild tend to fail in one of two directions: heavyweight platforms that lock your agent's brain behind their API, or notebook-grade scripts that evaporate on restart. I wanted a personal agent infrastructure with a different center of gravity:

1. The agent is a folder. A whole agent — identity, tools, knowledge, schedule, memory — should be inspectable and versionable as plain files and Python functions. No YAML schema that grows until it's a programming language, no external state I can't read.
2. Lifelong memory on a real budget. LLM context windows are lossy and paid for by the token. The agent needs to remember — verbatim recent context, semantic retrieval of the distant past — backed by a boring, reliable SQLite file per agent.
3. Capability without context rot. Real agents accumulate toolsets (research, finance tracking, documents, email, scheduling) far too large to keep in the system prompt at once. The host must let agents swap capability sets at runtime without losing identity or memory.
4. Presence. An agent that only exists when I open a terminal isn't an assistant. It needs interfaces where I already am (chat platforms), and autonomy (scheduled events, self-managed triggers) independent of any client.

The design bet: most agent frameworks over-abstract. If the unit of an agent is a folder, everything else — hosting, interfaces, memory, scheduling — becomes small, replaceable plumbing.

2. Solution Architecture

                    ┌─────────────────────────────┐
                    │         Agent (core)        │
                    │  OpenAI-compatible LLM loop │
                    └──────────────┬──────────────┘
           ┌───────────────┬───────┴────────┬───────────────────┐
           ▼               ▼                ▼                   ▼
   ┌──────────────┐ ┌─────────────┐ ┌──────────────┐  ┌───────────────┐
   │   Memory     │ │   Toolbox   │ │   Planner    │  │    Events     │
   │ SQLite + RAG │ │  discovery  │ │ DAG multi-   │  │ APScheduler   │
   │ per agent    │ │ (runtime    │ │ step plans,  │  │ (agent-managed│
   │              │ │ switching   │ │ step-by-step │  │  cron, YAML)  │
   └──────────────┘ └─────────────┘ └──────────────┘  └───────────────┘
           ▲               ▲                ▲                   ▲
           └───────────────┴───────┬────────┴───────────────────┘
                                   │ built-in + toolbox tools
                    ┌──────────────┴──────────────┐
                    │        Interfaces           │
                    │ Textual TUI · CLI (--once)  │
                    │ FastAPI + SSE server        │
                    │ Discord · Telegram · WhatsApp│
                    └─────────────────────────────┘

An agent is a directory:

agents/finance/
├── WHOAMI.md      # system prompt — the agent's identity
├── CRITICAL.md    # high-priority instructions injected above everything
├── agent.yaml     # model, memory, built-in tools, secrets, bridging config
├── tools/         # Python functions → LLM tool schemas (auto-discovered)
├── skills/        # Markdown knowledge snippets, loaded on demand via get_skill
└── events.yaml    # scheduled triggers (daily/interval/one-shot, human-readable)

Key design decisions:

• Toolbox switching instead of tool sprawl. Toolsets live in composable toolboxes — directory bundles of tools/ (functions) and skills/ (Markdown). A finance agent swaps to the researcher toolbox mid-conversation to browse the web, then switches back, keeping thread state and memory intact. Toolbox state is per-thread, so two conversations with the same agent can hold different toolsets. Only built-ins persist across switches.
• Skills as lazy-loaded Markdown. Skill summaries sit in the system prompt; full content is fetched via a get_skill built-in only when relevant. Knowledge scales without eating the context window.
• Secrets outside the codebase. API keys live in a KeePass .kdbx database, loaded into environment variables at runtime — nothing sensitive in agent.yaml or git.
• Model-agnostic. The openai SDK pointed at any OpenAI-compatible endpoint (default DeepInfra; Ollama/vLLM work for everything), with reasoning-effort and penalty controls.

3. Implementation

The agent loop (agent.py, ~1,100 lines)

Async OpenAI streaming loop with full tool-calling support: tool schemas are generated from Python functions by signature inspection, executed through a ToolRunner, and results fed back until the model produces a final answer. Context assembly layers, in order: agent identity (WHOAMI.md), critical instructions, skill summaries, active toolbox tools, current datetime grounding, then RAG-recalled memory. Streaming responses propagate to every interface via async generators.

Memory with RAG (memory.py, ~700 lines)

Per-agent SQLite database with three tiers of recollection:

• Recent window — recent messages stored verbatim, replayed into the prompt.
• Semantic retrieval — older conversation content is chunked, embedded, and retrieved by cosine similarity against the current query, so week-old context resurfaces exactly when relevant. Embeddings via bge-m3 through any OpenAI-compatible endpoint.
• Key/value store — for durable facts the agent explicitly wants to keep.

The retrieval pipeline is the same one exposed to the researcher toolbox: chunk → embed → top-k cosine similarity.

Planning (planning.py)

Agents can create structured multi-step plans as a DAG: plan(action="create") builds steps with dependencies and optional per-step toolbox assignment; plan(action="next") executes exactly one step as a focused agent.chat() invocation. Step results are persisted to a task-state database (with statuses: pending/running/completed/failed/skipped/blocked) and injected into later step prompts — so a ten-step task survives context limits, crashes, and thread switches.

Scheduled events (events.yaml + APScheduler)

Instead of cron syntax, schedules are structured YAML (daily, weekday, day-of-week, interval, one-shot; multiple times per trigger; timezone-aware via ZoneInfo) validated by Pydantic. Agents manage their own triggers through the event_tool CRUD tool, and the scheduler hot-reloads when the YAML changes. An agent can decide, mid-conversation, to remind itself to do something tomorrow.

Interfaces

• Textual TUI (chat_tui.py, ~1,400 lines): streaming chat, tool-call/result cards, thread picker, @path file attachment (with transparent text extraction from .docx/.pdf/.xlsx/.csv), and a chatless monitor mode for watching scheduled activity.
• CLI: one-shot queries (--once) for scripting, plain-text loop mode, thread monitoring.
• FastAPI + SSE server: streaming HTTP API for any client.
• Bridges: Discord (mention/prefix triggers, access control, attachments, message edit), Telegram (polling, access control), WhatsApp (Node.js/Baileys bridge with QR pairing).

The toolboxes

Shipped as working examples of the model: finance (18 tools — double-entry ledger, budgets with threshold checks, trend analysis, exchange rates, encrypted backup), researcher (DuckDuckGo search + crawl4ai fetching, chunked and embedded into a local vector store with bge-m3, budgeted query plans), document_writer (DOCX/XLSX/CSV/Markdown generation), gmail, and todo.

4. Results

• In daily personal use as my own assistant host — the finance, research, and document toolboxes run as real tools, not demos.
• One host, six interfaces, no per-agent glue code: the same folder-based agent is reachable from terminal, HTTP, and three chat platforms. Adding an agent is adding a folder.
• Memory that survives: SQLite + RAG retrieval recovers context far outside the context window — the agent references conversations from weeks ago when they're relevant, at zero marginal token cost for old messages.
• ~8,400 lines of core code doing what multi-thousand-line frameworks do, with every abstraction earned rather than speculative.

What I took from it

The interesting engineering in agents isn't the model call — it's everything around it: what to put in the system prompt, what to retrieve and when, how to bound a tool's blast radius, and how to give an agent agency (schedules, planning, toolbox switching) without giving it fragility. Building the host taught me more about where LLM agents actually fail than any framework tutorial could — and the framework-sized lessons (bounded toolsets, validated schemas, durable memory) are exactly what Automation Forge optimizes next.

Treat the agent — its prompt and tool schemas — as an optimizable artifact, and close the loop with no human inside it.

You have an agent — a system prompt and a set of tool definitions — and a fixed set of test cases with expected outputs. Automation Forge runs the loop for you: execute every case, judge the failures, diagnose why they failed, apply targeted edits to the prompt and tool descriptions, and repeat — until quality thresholds are met or a budget is spent. No human in the loop.

Repo: github.com/titut/automation-forge · Born from: AgentHost

1. Problem Statement

While running live agents on AgentHost, the bottleneck stopped being the model and became the configuration: a system prompt with one ambiguous sentence, or a tool description with one ambiguous parameter, would produce large, inconsistent variance in output quality. The standard fix — hand-editing the prompt, rerunning tests, eyeballing the difference — is slow, unrepeatable, and throws away the reason each edit worked.

The problem statement: given a starting configuration (system prompt + tool schemas) and a fixed set of user-provided test cases, autonomously produce a configuration that passes them reliably.

Three constraints shaped the entire design:

1. No human in the loop. The user provides inputs and the oracle (expected results) up front; everything between is autonomous. Human review happens before and after, not during.
2. The optimizer must not be able to break what it's fixing. The obvious failure mode of prompt-optimization loops is a rewrite-happy editor that improves one case while silently regressing three others. Edits must be structured and permissioned, not free-text.
3. A pass is not a pass. LLMs are non-deterministic. A configuration that scores 100% once is not proven — the loop must demand statistical evidence before declaring success.

2. Solution Architecture

 User provides (once):                     The optimization loop:
 ┌─────────────────────┐
 │ system_prompt.md    │        ┌──────────────────────────────────────┐
 │ tool_defs.json      │        │  ① Test Harness                      │
 │ cases.json          │───▶    │     run config against every case    │
 │ config.yaml         │        │     → output + reasoning + tool calls│
 └─────────────────────┘        └──────────────┬───────────────────────┘
                                               ▼
                                ┌──────────────────────────────────────┐
                                │  ② Evaluator (LLM)                   │
                                │     pass/fail → failure class from   │
                                │     a fixed taxonomy → diagnosis     │
                                │     → proposed structured edits      │
                                └──────────────┬───────────────────────┘
                                               ▼
                                ┌──────────────────────────────────────┐
                                │  ③ Loop Controller                   │
                                │     aggregate scores · check budgets │
                                │     stop thresholds · pick failures  │
                                └──────────────┬───────────────────────┘
                                               ▼
                                ┌──────────────────────────────────────┐
                                │  ④ Editor (permission-gated)         │
                                │     6 whitelisted operations,        │
                                │     validated → new version          │
                                └──────────────┬───────────────────────┘
                                               ▼
                                ┌──────────────────────────────────────┐
                                │  ⑤ Version Store                     │
                                │     every config + scores persisted, │
                                │     best-so-far always recoverable   │
                                └──────────────────────────────────────┘

Design principles (from SPEC.md, enforced in code):

The six edit operations

The evaluator proposes edits, but can only speak in this grammar — each permission-gated per configuration:

1. edit_system_prompt — replace the prompt or one marked section
2. add_constraint — append a rule
3. edit_tool_description — rewrite a tool's top-level description
4. edit_parameter_description — rewrite one parameter's description
5. rename_tool_display — change display name, never the function binding
6. edit_tool_schema — structural change, disabled by default

Every edit carries a reason string and is validated (Pydantic + permission check) before application. Notably, most gains come from the cheap, safe edits: clarifying a tool description or parameter docstring fixes tool-misuse failures without touching the prompt at all.

3. Implementation

The evaluator: a doctor, not a grader

The evaluator LLM receives the full context — current prompt, tool schemas, the case input, the expected-result spec, the agent's actual output, its reasoning trace, its tool calls — and must return strict JSON: pass/fail, a failure class from a fixed taxonomy (missing_step, wrong_format, tool_misuse, hallucination, ignored_constraint, ambiguous_instruction, missing_context, over_complexity), a diagnosis citing specific wording, and ≥ 1 proposed edit for every failing case (enforced — an empty edit list on a failure is invalid output).

Scoring without an LLM oracle (metrics.py)

Deterministic scoring keeps the ground truth trustworthy: exact (normalized text), contains / contains_all (partial credit), json_match (parse + path check), and custom evaluators. The weighted score combines case pass rates with per-case weights — so the loop optimizes what the user actually specified, judged by code, not by another LLM's opinion.

Regression gating — the part most optimizers skip

This is the feature I'd keep if I kept only one:

• A configuration that reaches the single-run thresholds isn't accepted yet. Each test case is re-sampled samples_per_case times.
• Success requires both the overall pass rate and every individual case's pass rate to clear configured minimums (min_pass_rate, min_case_pass_rate).
• If regression fails, the loop continues optimizing — and the regression results are persisted (version_NNN_regression.json) as evidence.

This directly attacks the non-determinism constraint: "it passed once" is not a claim this system is willing to make. The same gate is available standalone via forge regression <run> --samples N — a post-hoc confidence check for any configuration, optimized by Forge or by hand.

Runs are durable artifacts

Every run is a directory with a manifest.json (version history, best-version id, status), per-version case results, and regression records. forge resume continues an interrupted run; forge show-best recovers the winning configuration; budgets mean an unattended overnight run can't bill a surprise. The CLI (forge run / optimize / resume / show-best / regression) is built on click with rich output; every role (agent/evaluator/editor) can point at a different OpenAI-compatible endpoint.

Where it came from

Automation Forge exists because AgentHost made the failure modes visible: watching real agents misuse tools because a description was vague, or ignore constraints buried mid-prompt. The taxonomy in the evaluator is literally the list of failures I watched live agents commit. It optimizes AgentHost-style configurations — which is why tool_defs.json is plain OpenAI tool-schema format and tool implementations are ordinary Python TOOL_HANDLERS.

4. Results

• The loop works end to end, unattended: on the bundled research_task example, an autonomous run took a configuration that failed cases, diagnosed them (tool misuse and format issues), applied targeted edits — and reached the full pass threshold, with the regression gate then re-sampling every case before declaring success. Real run artifacts (manifests, per-case results, regression records) ship in the repo.
• Structured edits demonstrably outperform free-text rewrites in the failure modes that matter: diagnosing tool_misuse and fixing one parameter description is a surgical, reviewable change — something hand prompt-tuning rarely achieves.
• Model-agnostic by construction: the whole loop runs against local endpoints (Ollama/vLLM), making optimization iterations cheap and private.
• Built tested: harness, evaluator, editor permissions, regression gating, and fallback edits all have dedicated test modules.

Honest scope

The current evaluator is single-pass per case, and the oracle modes are exact/match-based rather than rubric-based — "did the agent actually reason well" is out of scope by design (the user owns the oracle). The roadmap is about statistical confidence, not capability: more samples, per-class diagnosis depth, and smarter failure-selection heuristics for the next iteration's edits.

What I took from it

Evals are not a step after building an agent — they're the only way to change an agent without superstition. After building this, I can't unsee it: every prompt tweak is an uncontrolled experiment unless there's a fixed case set, a deterministic judge, and a regression gate. The loop is small; the discipline it enforces is the whole game.