Building AgentOS AI: Multi-Agent Observability with OpenTelemetry and SigNoz

 

Building AgentOS AI: End-to-End Observability for Multi-Agent AI using OpenTelemetry and SigNoz

Author: Geet Bharambe

 

 

Introduction

Artificial Intelligence is becoming increasingly capable, but as AI systems grow more complex, they also become harder to understand and debug. Modern AI applications rarely consist of a single model call—instead, they involve multiple specialized agents working together to solve complex problems.

During the WeMakeDevs Agents of SigNoz Track 1 Hackathon, I wanted to build something that demonstrates not only how multiple AI agents collaborate, but also how developers can observe everything happening inside those agents.

This idea led to AgentOS AI, a real-time multi-agent AI observability platform built using Google Gemini, FastAPI, OpenTelemetry, and SigNoz.

The project's goal is simple:

Make AI agent execution transparent, measurable, and easy to debug.

The Problem

Most AI applications behave like black boxes.

When a response is incorrect, slow, or expensive, developers often have no idea:

  • Which agent caused the delay?
  • Which step failed?
  • How many tokens were consumed?
  • How much did the request cost?
  • Which retry actually succeeded?
  • Where did latency increase?

Traditional logging isn't enough to answer these questions.

This is where observability becomes essential.

Project Overview

 

AgentOS AI is a collaborative AI system consisting of four specialized agents:

  • Manager
  • Researcher
  • Coder
  • Writer

Each agent performs a dedicated responsibility within a coordinated workflow.

Instead of executing independently, the agents collaborate by passing information through the pipeline.

Multi-Agent Workflow

 

The execution flow follows these stages:

Step 1 – Manager

The Manager receives the user's request.

Instead of answering directly, it:

  • Understands the task
  • Breaks it into smaller steps
  • Creates an execution plan

Step 2 – Researcher & Coder

Using the Manager's plan:

  • Researcher gathers technical knowledge
  • Coder generates implementation

Both execute simultaneously to reduce latency.

Step 3 – Writer

The Writer consumes the actual outputs from both agents.

Instead of receiving the original prompt again, it synthesizes:

  • Research findings
  • Technical implementation

into one structured response.

Step 4 – Manager Review

Finally, the Manager performs a final review and produces the polished response shown to the user.

This creates a genuine collaborative AI workflow rather than multiple independent LLM calls.

Technology Stack

The project was built using:

  • Python
  • FastAPI
  • Google Gemini
  • OpenTelemetry
  • SigNoz
  • HTML
  • CSS
  • JavaScript

Why I Chose SigNoz

The focus of this hackathon was AI observability.

Instead of only displaying the final response, I wanted complete visibility into the entire execution process.

SigNoz provided exactly that.

Using OpenTelemetry, every important event inside the pipeline is exported directly into SigNoz.

How I Used OpenTelemetry

Every agent execution creates its own trace span.

Examples include:

  • Manager planning
  • Research execution
  • Code generation
  • Documentation synthesis
  • Final review

Each span records useful attributes including:

  • Model name
  • Token usage
  • Cost
  • Execution latency
  • Pipeline status

When an error occurs, the span automatically records:

  • Exception details
  • Error status
  • Retry information

This makes debugging significantly easier.

What I Monitor in SigNoz

Distributed Traces

 

Each pipeline execution generates a complete distributed trace.

Developers can see:

  • execution order
  • parent-child relationships
  • duration of every agent

Logs

 

Structured logs capture:

  • agent start
  • completion
  • retries
  • failures
  • pipeline status

These logs make debugging much faster.

Metrics

 

The application records metrics such as:

  • token usage
  • latency
  • API cost
  • execution counts

These metrics help evaluate both performance and operational cost.

Cost Tracking

One feature I wanted to include was accurate API cost monitoring.

Instead of estimating usage, the application calculates cost using actual Gemini token usage returned by the API.

The project tracks:

  • input tokens
  • output tokens
  • estimated API cost

This allows developers to understand the financial impact of each request.

Error Handling

Observability becomes valuable when failures occur.

The project includes:

  • automatic retry logic
  • exception recording
  • error spans
  • failure logging

Retries only happen for genuine API failures.

No artificial failures are generated.

This ensures the telemetry shown in SigNoz accurately reflects real system behavior.

Challenges I Faced

Building the project involved several challenges.

Designing a True Multi-Agent Pipeline

Initially, each agent processed the same prompt independently.

This wasn't real collaboration.

The architecture was redesigned so that:

Manager → Researcher & Coder → Writer → Manager Review

creating genuine data flow between agents.

Accurate Telemetry

Another challenge was ensuring that telemetry reflected actual execution.

This required:

  • proper OpenTelemetry instrumentation
  • accurate span attributes
  • correct pipeline status
  • truthful UI state
  • real retry behavior

Model Migration

During development, Google updated Gemini models.

The application was migrated to the newer model while maintaining observability features and updating cost calculations.

What I Learned

This project taught me that building AI applications is only part of the challenge.

Understanding how those systems behave in production is equally important.

Observability provides visibility into:

  • performance
  • reliability
  • cost
  • debugging
  • user experience

Without telemetry, diagnosing issues becomes difficult.

With OpenTelemetry and SigNoz, every stage becomes measurable.

Future Improvements

If I continue developing AgentOS AI, I would like to add:

  • Dynamic model selection based on latency
  • Historical execution analytics
  • User authentication
  • Agent memory
  • Additional specialized agents
  • Alerting rules inside SigNoz
  • Dashboard customization

Conclusion

AgentOS AI demonstrates how observability can be integrated directly into modern AI workflows.

Rather than treating monitoring as an afterthought, the project instruments every stage of execution using OpenTelemetry and visualizes it in SigNoz.

Developers gain visibility into traces, metrics, logs, latency, token usage, API cost, retries, and failures—all in one place.

Building this project helped me better understand both AI systems and production-grade observability, and it reinforced how valuable telemetry is for building reliable AI applications.

GitHub Repository

GitHub:
https://github.com/Geet-Bharambe/agentos-ai

Demo Video

YouTube:

 https://youtu.be/4rinTetA_O8

 

Comments

Post a Comment