Conduit: A UI-less node-based system

This is a submission for the Amazon Q Developer "Quack The Code" Challenge: Exploring the Possibilities

What I Built

I have built Conduit, a domain-specific language (DSL) for creating node-based workflows. Conduit enables developers to create reusable building blocks with an intuitive syntax that can be mixed and matched to build complex data processing pipelines. It's similar to node-based UI tools like ComfyUI but without the graphical interface, offering a code-first approach that's more flexible and embeddable.

One of Conduit's most powerful features is its true cross-language compatibility. Unlike many workflow tools that are tied to specific languages or platforms, Conduit workflows can be compiled to native libraries (.so/.dll) and seamlessly integrated with virtually any programming language through FFI (Foreign Function Interface). This means you can define your workflows once in Conduit's intuitive DSL and then use them from JavaScript, Python, PHP, Java, Go, or any other language that supports FFI - all while maintaining native performance. This language-agnostic approach eliminates the need for complex interoperability layers or performance-degrading bridges between different technology stacks.

What makes this project special is how Amazon Q Developer transformed my development process. With its assistance, I was able to build in TWO days what would have taken a month otherwise, including creating a custom DSL parser, implementing an ECS architecture, and developing procedural macros — all challenging aspects of Rust development that typically require significant time investment and expertise.

Demo

The project includes working examples that demonstrate Conduit's capabilities:

• A Rust example that reads an image file, resizes it, and saves the result:

let pipeline = r#"
  resizer {
    source <- read_file {
      input <- "./examples/conduit-example/cover.png"
    }
    width <- 512
    height <- 215
    output -> write_file {
      destination <- "./examples/conduit-example/cover.smaller.png"
    }
  }
"#;

let mut engine = Engine::new();
engine.run_pipeline(pipeline);

If you're a visual learner, the above example can be visualized like this:

Demo code can be found here.

• A Node.js integration example showing how Conduit can be used from JavaScript:

import { ConduitEngine } from './Engine'

const engine = new ConduitEngine()

const pipeline = `
  resizer {
    source <- read_file {
      input <- "../conduit-example/cover.png"
    }
    width <- 512
    height <- 215
    output -> write_file {
      destination <- "../conduit-example/cover.smaller.png"
    }
  }
`

engine.runPipeline(pipeline)
engine.destroy()

Demo code can be found here.

Code Repository

[dependencies]
conduit = { git = "https://github.com/milewski/conduit-challange.git", version = "0.1.0" }
conduit-derive = { git = "https://github.com/milewski/conduit-challange.git", version = "0.1.0" }

use conduit::Engine;
use conduit_derive::Node;
#[derive(Node)]
pub struct Logger {
    pub left: Input<String>,
    pub right: Input<String>,
}

impl ExecutableNode for Logger {
    fn run(&self) {
        println!("{} {}", self.left.read

I intend to grow this into an open-source project because deep inside, this is ideally how I would like ComfyUI to be. There's still a long journey ahead for building all the custom nodes, which is especially challenging given that the majority of code for AI workflows is written in Python. However, with my hands-on experience with Candle and Burn libraries, I may be able to get pretty close!

The Problem and Inspiration

I'm a huge fan of ComfyUI, a powerful node-based system I’ve used extensively in hackathons and personal projects. But despite its strengths, I kept running into two consistent limitations:

1. When I needed functionality beyond what the existing nodes or community contributions provided, I had to build it myself — typically in Python, through a manual and often clunky process.

2. Systems like ComfyUI are not easy to embed. Sometimes, I just want a lightweight, embeddable engine that works directly within the language I'm using — without setting up servers, UIs, or dealing with platform-specific quirks.

This led me to a question:

What if there were a tool like ComfyUI, but entirely code-driven — no GUI, no friction, just flexible building blocks?

What if I could create node-based workflows not just for ML models, but for anything — like making API calls, evaluating expressions, chaining CLI commands, validating form data, or even just adding numbers?

That’s the idea behind Conduit: a node-based system that’s completely UI-less, embeddable, and designed for developers who want full control without leaving their editor.

Technical Implementation

Conduit features a simple, intuitive syntax heavily inspired by hardware description languages like Verilog. Creating custom nodes is straightforward—just define a Rust struct, derive it with the Node macro, and you're ready to build. No need to deal with Python or JavaScript; everything is native Rust with the ability to be called from other languages via FFI.

I wrote a basic tutorial explaining the syntax here, which covers everything you need to know about the DSL: https://github.com/milewski/conduit-challenge?tab=readme-ov-file#basic-syntax

One of the key advantages of Conduit is its cross-language compatibility. You can build your workflow, compile it to a .so/.dll dynamic library, and use it in ANY programming language via FFI. I've provided an example of how to use it in JavaScript, but there are no limitations—you could integrate it with Dart, PHP, Java, or any other language. Because the workflows run natively, you get native performance without the bottlenecks specific to each language's runtime environment.

Tech Stack

The project was built entirely in Rust, with a custom DSL written using PEST, a powerful parsing expression grammar library. The language grammar (PEG) can be found in schema.pest.

Performance is a core advantage of Conduit's architecture. By leveraging Rust's zero-cost abstractions and the ECS pattern, Conduit workflows execute with near-native performance regardless of complexity. Unlike traditional scripting-based workflow systems that interpret code at runtime or rely on language-specific VMs, Conduit compiles workflows to optimized machine code. This approach eliminates interpreter overhead and enables automatic parallelization of independent nodes. In practical terms, this means Conduit can process data-intensive workflows (like image manipulation or large dataset transformations) significantly faster than equivalent implementations in interpreted languages, while maintaining predictable memory usage and avoiding garbage collection pauses.

Conduit adopts an Entity Component System (ECS) design pattern using bevy-ecs, which makes it simple to parallelize operations where possible, resulting in extremely efficient workflow execution. It also utilizes an inventory system to automatically register nodes, further enhancing the ergonomics of the project.

The project is divided into two crates:

• Conduit: The main crate responsible for parsing the DSL and executing the node workflows
• Conduit-Derive: A procedural macro crate that eliminates boilerplate code when creating new nodes

How I Used Amazon Q Developer

Before this hackathon, I wasn't aware that Amazon Q CLI existed. Initially, I assumed it would be similar to other IDE tools like GitHub Copilot or Gemini Assistant—primarily code auto-completion tools. However, I was pleasantly surprised to discover it was a full-featured CLI assistant, my first experience using AI directly from the command line.

Development Journey with Amazon Q

My journey began by asking Amazon Q for hackathon project ideas. I provided context about my interest in node-based systems like ComfyUI, and it suggested building something similar to what Conduit has become. When I asked for a prototype, the quality of the output immediately gave me confidence that I was working with a powerful tool.

The code Amazon Q produced was remarkably similar to what I would have written myself, which gave me the confidence to continue. What would have taken me months to build, I was able to accomplish in just two days with Amazon Q's assistance!

Another point worth mentioning: I saw there was extensive documentation on the AWS Q website, but I skipped it all and jumped straight to the installation. The command line itself has a /help command that explains what features are available, making it effortless to figure out how to use the tool. This ease of use was a pleasant surprise.

Architectural Decisions

I started from scratch, first creating the foundation and then asking Amazon Q to help build feature by feature. For straightforward tasks, I wrote the code myself, but for more complex challenges that would typically require significant time to figure out the best approach, I asked Amazon Q to provide multiple options.

What impressed me most was Amazon Q's ability to generate substantial amounts of code while maintaining a step-by-step approach. I could discuss concerns like, "I think if you change the struct this way, it would require a lifetime parameter and could cause trouble when implementing X Y Z" and it would understand my concerns and propose alternative approaches that aligned with the existing codebase.

Advanced Rust Features

Once my first version was working, I wanted to simplify the codebase to reduce boilerplate and create a more ergonomic library. That's when I asked Amazon Q to create a procedural macro to generate all the necessary boilerplate. Amazon Q condensed it into a single #[derive(Node)] attribute and made the code super clean to use! Admittedly, procedural macros in Rust are hard to write and would have taken me a long time to implement myself, but with a single prompt, Amazon Q was able to build it and make it work on the first try!

I also discussed with Amazon Q what would be the best design patterns to apply to this project for better performance, parallel execution, and multi-threading. Initially, I implemented it using channels, but Amazon Q recommended using the ECS pattern instead, which made the implementation much simpler. This guidance on architectural decisions was invaluable.

Tips for Using Amazon Q Developer

• Work incrementally: Ask Amazon Q to write or change code in small increments rather than tackling large features all at once. This isn't because it can't handle complex tasks, but because smaller changes are easier to understand and review. You won't be able to effectively follow along, make changes, or fix bugs if you don't understand the code being produced.

• Commit frequently: Always commit your code before asking Amazon Q to make changes. Even though it shows and confirms changes before implementing them, having a git history makes it easier to understand what was modified and where.

• Provide clear direction: Give Amazon Q hints about what you want to accomplish. Specify which files you want to modify, what changes you want to make, and how they should be implemented. It works best when you provide specific file and function names.

• Create an AmazonQ.md file: Write a guide explaining what Amazon Q should and shouldn't do, along with project context. Including instructions to always run tests or compile the project helps ensure that Amazon Q will verify its changes and fix any errors before finalizing them.

Future Development

If you like the project and want to help it move forward, contributions are welcome! There are several features and improvements on the roadmap:

• Async Runtime/Scheduler: Implement support for nodes that can run asynchronously
• Enhanced Error Handling: Complete the integration with thiserror for more robust error management
• Arithmetic Expressions: Update the DSL to support arithmetic operations:

  node { property <- (500 * 5) / (10 * another::module) }

• Conditional Logic: Add support for conditional nodes:

  test { property <- if (true && false, "when-true", "when-false") } 

• Expanded Node Library: Implement more nodes for various use cases

Pull requests addressing any of these areas would be greatly appreciated!