Skip to content

feat(examples): add Microsoft Agent Framework tracing example #667

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
viniciusdsmello wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(examples): add Microsoft Agent Framework tracing example #667

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
315 changes: 315 additions & 0 deletions examples/tracing/microsoft-agent-framework/microsoft_agent_framework_tracing.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,315 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "edd15f6e",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openlayer-ai/openlayer-python/blob/main/examples/tracing/microsoft-agent-framework/microsoft_agent_framework_tracing.ipynb)\n",
"\n",
"\n",
"# <a id=\"top\">Microsoft Agent Framework quickstart</a>\n",
"\n",
"This notebook shows how to export traces captured by [Microsoft Agent Framework](https://learn.microsoft.com/en-us/agent-framework/) (MAF) to Openlayer. The integration is done via the Openlayer's [OpenTelemetry endpoint](https://www.openlayer.com/docs/integrations/opentelemetry).\n",
"\n",
"Microsoft Agent Framework is the successor to Semantic Kernel and AutoGen, which are both in maintenance mode. It is natively instrumented with OpenTelemetry, so no extra instrumentation library is needed.\n",
"\n",
"This notebook was written against `agent-framework` 1.14. The observability API changed in the 1.x line \u2014 earlier releases used `enable_instrumentation(enable_sensitive_data=True)` where current ones use `enable_sensitive_telemetry()`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6f6d7b64",
"metadata": {},
"outputs": [],
"source": [
"!pip install agent-framework opentelemetry-sdk opentelemetry-exporter-otlp-proto-http"
]
},
{
"cell_type": "markdown",
"id": "72773c48",
"metadata": {},
"source": [
"## 1. Set the environment variables\n",
"\n",
"Agent Framework ships with the OpenTelemetry **API** only. It does not bundle an exporter and does not read a `.env` file on its own, so you configure both yourself.\n",
"\n",
"Two details matter for Openlayer:\n",
"\n",
"- **Protocol.** Agent Framework defaults to OTLP over gRPC. Openlayer's endpoint speaks OTLP over HTTP, so `OTEL_EXPORTER_OTLP_PROTOCOL` must be set to `http/protobuf` or nothing arrives.\n",
"- **Traces-only.** Using the signal-specific `OTEL_EXPORTER_OTLP_TRACES_*` variables means only a trace exporter is created. If you set the generic `OTEL_EXPORTER_OTLP_ENDPOINT` instead, Agent Framework also builds metric and log exporters and points them at endpoints Openlayer does not serve."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bf295680",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"os.environ[\"OPENAI_API_KEY\"] = \"YOUR_OPENAI_API_KEY_HERE\"\n",
"\n",
"# Env variables pointing to Openlayer's OpenTelemetry endpoint\n",
"os.environ[\"OTEL_EXPORTER_OTLP_TRACES_ENDPOINT\"] = \"https://api.openlayer.com/v1/otel/v1/traces\"\n",
"os.environ[\"OTEL_EXPORTER_OTLP_TRACES_HEADERS\"] = \"Authorization=Bearer YOUR_OPENLAYER_API_KEY_HERE, x-bt-parent=pipeline_id:YOUR_OPENLAYER_PIPELINE_ID_HERE\"\n",
"os.environ[\"OTEL_EXPORTER_OTLP_PROTOCOL\"] = \"http/protobuf\"\n",
"\n",
"# Agent Framework captures no message content by default: prompts, completions and\n",
"# tool arguments/results are all omitted unless you opt in. Microsoft discourages\n",
"# enabling this in production, since it puts user data in your telemetry backend.\n",
"# Without it, traces arrive looking healthy but carry nothing to evaluate.\n",
"os.environ[\"ENABLE_SENSITIVE_DATA\"] = \"true\""
]
},
{
"cell_type": "markdown",
"id": "66e5ddfd",
"metadata": {},
"source": [
"## 2. Configure the OpenTelemetry providers\n",
"\n",
"`configure_otel_providers()` reads the standard `OTEL_EXPORTER_OTLP_*` variables and wires up the trace provider and exporter for you."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "534b5079",
"metadata": {},
"outputs": [],
"source": [
"from agent_framework.observability import configure_otel_providers\n",
"\n",
"configure_otel_providers()"
]
},
{
"cell_type": "markdown",
"id": "20fa3880",
"metadata": {},
"source": [
"If you already configure OpenTelemetry yourself \u2014 with the Azure Monitor distro, an OTel Collector, or your own `TracerProvider` \u2014 skip `configure_otel_providers()` entirely. Agent Framework's instrumentation is **on by default** and will emit into whatever providers are globally registered. In that case you only need to opt in to message content:\n",
"\n",
"```python\n",
"from agent_framework.observability import enable_sensitive_telemetry\n",
"\n",
"enable_sensitive_telemetry()\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "b44caf6f",
"metadata": {},
"source": [
"## 3. Use agents as usual\n",
"\n",
"That's it! Now you can continue using agents and tools as usual. The trace data is automatically exported to Openlayer and you can start creating tests around it."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "58d8324e",
"metadata": {},
"outputs": [],
"source": [
"from typing import Annotated\n",
"\n",
"from pydantic import Field\n",
"from agent_framework import Agent, tool\n",
"from agent_framework.openai import OpenAIChatClient\n",
"\n",
"\n",
"@tool(approval_mode=\"never_require\")\n",
"def get_weather(\n",
" location: Annotated[str, Field(description=\"The location to get the weather for.\")],\n",
") -> str:\n",
" \"\"\"Get the current weather for a given location.\"\"\"\n",
" return f\"The weather in {location} is sunny with a high of 21C.\"\n",
"\n",
"\n",
"agent = Agent(\n",
" client=OpenAIChatClient(model=\"gpt-4o-mini\"),\n",
" name=\"WeatherAgent\",\n",
" instructions=\"You are a helpful weather assistant. Use the tools available to you.\",\n",
" tools=get_weather,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0331bc0f",
"metadata": {},
"outputs": [],
"source": [
"result = await agent.run(\"What's the weather like in Seattle?\")\n",
"result.text"
]
},
{
"cell_type": "markdown",
"id": "ed67318a",
"metadata": {},
"source": [
"This produces an `invoke_agent WeatherAgent` trace containing a `chat` span per model call and an `execute_tool get_weather` span, with prompts, completions, token counts and cost.\n",
"\n",
"> **Do not wrap your run in a root span.** Agent Framework's own samples open a manual parent span to print a trace ID for Application Insights. On Openlayer that costs you the record's output: Openlayer builds a record from the root span, and a plain span carries no output, so the record arrives empty even though the content is present one level down. Let `invoke_agent` be the root and the record gets the full exchange."
]
},
{
"cell_type": "markdown",
"id": "3db38bc7",
"metadata": {},
"source": [
"## 4. Trace a graph workflow\n",
"\n",
"Agent Framework's workflow engine emits its own spans \u2014 `workflow.build`, `workflow.run`, `executor.process`, `edge_group.process` and `message.send` \u2014 outside the GenAI semantic conventions.\n",
"\n",
"The workflow below routes a support ticket to one of two executors through conditional edges."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8a06829d",
"metadata": {},
"outputs": [],
"source": [
"from typing_extensions import Never\n",
"\n",
"from pydantic import BaseModel\n",
"from agent_framework import Executor, WorkflowBuilder, WorkflowContext, handler\n",
"\n",
"\n",
"class Ticket(BaseModel):\n",
" text: str\n",
" priority: str = \"unknown\"\n",
"\n",
"\n",
"class TriageExecutor(Executor):\n",
" \"\"\"Assign a priority to the incoming ticket.\"\"\"\n",
"\n",
" @handler\n",
" async def triage(self, text: str, ctx: WorkflowContext[Ticket]) -> None:\n",
" priority = \"urgent\" if \"outage\" in text.lower() else \"routine\"\n",
" await ctx.send_message(Ticket(text=text, priority=priority))\n",
"\n",
"\n",
"class EscalateExecutor(Executor):\n",
" \"\"\"Handle urgent tickets.\"\"\"\n",
"\n",
" @handler\n",
" async def escalate(self, ticket: Ticket, ctx: WorkflowContext[Never, str]) -> None:\n",
" await ctx.yield_output(f\"ESCALATED: {ticket.text}\")\n",
"\n",
"\n",
"class AutoReplyExecutor(Executor):\n",
" \"\"\"Handle routine tickets.\"\"\"\n",
"\n",
" @handler\n",
" async def auto_reply(self, ticket: Ticket, ctx: WorkflowContext[Never, str]) -> None:\n",
" await ctx.yield_output(f\"AUTO-REPLIED: {ticket.text}\")\n",
"\n",
"\n",
"triage = TriageExecutor(id=\"triage\")\n",
"escalate = EscalateExecutor(id=\"escalate\")\n",
"auto_reply = AutoReplyExecutor(id=\"auto_reply\")\n",
"\n",
"workflow = (\n",
" WorkflowBuilder(start_executor=triage)\n",
" .add_edge(triage, escalate, condition=lambda t: t.priority == \"urgent\")\n",
" .add_edge(triage, auto_reply, condition=lambda t: t.priority == \"routine\")\n",
" .build()\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "36cdaad2",
"metadata": {},
"outputs": [],
"source": [
"outputs = []\n",
"async for event in workflow.run(\"How do I reset my password?\", stream=True):\n",
" if event.type == \"output\":\n",
" outputs.append(event.data)\n",
"\n",
"outputs"
]
},
{
"cell_type": "markdown",
"id": "5fe76db2",
"metadata": {},
"source": [
"Running this notebook end to end produces **three** records, not one. Agent Framework opens a separate root span per top-level operation and never wraps them in a shared parent, so Openlayer sees three traces:\n",
"\n",
"| Record | Span | Emitted by |\n",
"| --- | --- | --- |\n",
"| the agent run, with content and tokens | `invoke_agent WeatherAgent` | `agent.run()` |\n",
"| a few milliseconds, no content | `workflow.build` | `WorkflowBuilder.build()` |\n",
"| the workflow execution, no content | `workflow.run` | `workflow.run()` |\n",
"\n",
"`workflow.build` is emitted when the graph is constructed, not when it runs. In a real application you build a workflow once at startup and run it many times, so you get one `workflow.build` record per process rather than one per request.\n",
"\n",
"The workflow itself arrives as its own record, with the executor and edge-group spans nested under `workflow.run`. They are stored as generic steps today, and their attributes are kept under `Other fields`.\n",
"\n",
"The edge that did *not* fire is the interesting one \u2014 every routing decision is recorded, including the messages that were silently dropped:\n",
"\n",
"```json\n",
"{\n",
" \"delivered\": false,\n",
" \"delivery_status\": \"dropped condition evaluated to false\",\n",
" \"id\": \"SingleEdgeGroup/6b5a7d2c-939e-4649-9a83-b829257b3ced\",\n",
" \"type\": \"SingleEdgeGroup\"\n",
"}\n",
"```\n",
"\n",
"Two things to keep in mind about workflow spans:\n",
"\n",
"- `ENABLE_SENSITIVE_DATA` does **not** apply to them. It gates the agent, chat and tool code paths only, so workflow spans carry structural metadata whether it is on or off. To capture executor inputs and outputs, read the `executor_invoked` and `executor_completed` events off `workflow.run(..., stream=True)` and record them yourself.\n",
"- Causality between a `message.send` span and the `executor.process` span it feeds is expressed with OpenTelemetry **span links** rather than parent/child nesting, because executors are not nested in one another. The tree still renders correctly \u2014 every span is a child of `workflow.run` \u2014 but the link edges are not shown."
]
},
{
"cell_type": "markdown",
"id": "d568c146",
"metadata": {},
"source": [
"## Notes\n",
"\n",
"- **Running outside a notebook.** Agent Framework does not auto-load `.env`. Call `load_dotenv()` yourself before configuring the providers, or pass `configure_otel_providers(env_file_path=\".env\")`.\n",
"- **Duplicate content.** If you instrument the chat client *and* the agent, prompts and responses are captured twice.\n",
"- **MCP tools.** Agent Framework injects `traceparent` into `tools/call` for MCP sessions your process opens (`MCPStreamableHTTPTool`, `MCPStdioTool`, `MCPWebsocketTool`). It cannot do so for hosted connectors such as `OpenAIChatClient.get_mcp_tool(...)`, where the provider runtime issues the call \u2014 traces stop at that boundary. Use a client-opened transport if you need end-to-end tracing."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}