Mastra Example with Reflight

This example demonstrates how to integrate Reflight with Mastra AI agents using OpenTelemetry (OTEL) tracing.

📦 Package Version: This documentation is written for @reflight/mastra version 0.20.0. If you encounter issues, ensure you're using this version or check the changelog (opens in a new tab) for breaking changes.

Installation

Install the Reflight SDK for Mastra:

npm install @reflight/mastra@0.20.0

⚠️ Version Compatibility: This example uses @reflight/mastra@0.20.0. If you're using a different version, some APIs may differ. Always check the version-specific documentation for your SDK version.

Mastra Integration

Follow these steps to integrate Reflight with your Mastra AI agents.

Step 1: Get Your API Key

Go to your project page and get your API key. Add it to your environment variables as REFLIGHT_API_KEY.

REFLIGHT_API_KEY=your_reflight_api_key

💡 Tip: Keep your API key secure and never commit it to version control. Use environment variables or a secrets manager.

Step 2: Configure Mastra Telemetry

Follow the Mastra OTEL Tracing documentation (opens in a new tab). When you reach the telemetry configuration section, use the following configuration:

📝 Note: Make sure you have @mastra/core installed. The Reflight exporter integrates seamlessly with Mastra's built-in OpenTelemetry support.

import { Mastra } from "@mastra/core/mastra";
import { createReflightExporter } from "@reflight/mastra";
 
export const mastra = new Mastra({
  observability: {
    configs: {
      otel: {
        serviceName: "demo-app",
        exporters: [
          createReflightExporter({ apiKey: process.env.REFLIGHT_API_KEY! }),
        ],
      },
    },
  },
  agents: {
    /* your agents */
  },
});

Complete Example

Here's a complete example showing the full Mastra setup with Reflight:

import { Mastra } from "@mastra/core/mastra";
import { createReflightExporter } from "@reflight/mastra";
import { publisherAgent } from "./agents/example-publisher-agent";
import { copywriterAgent } from "./agents/example-copywriter-agent";
import { editorAgent } from "./agents/example-editor-agent";
 
if (!process.env.REFLIGHT_API_KEY) {
  throw new Error("REFLIGHT_API_KEY is not set");
}
 
export const mastra = new Mastra({
  observability: {
    configs: {
      otel: {
        serviceName: "demo-app",
        exporters: [
          createReflightExporter({ apiKey: process.env.REFLIGHT_API_KEY! }),
        ],
      },
    },
  },
  agents: { copywriterAgent, editorAgent, publisherAgent },
});

What Gets Tracked

With Reflight configured, all Mastra operations are automatically tracked:

Agent Invocations: Every agent call with inputs and outputs
Tool Usage: All tool executions with parameters and results
Workflow Execution: Complete workflow traces
Performance Metrics: Latency, token usage, and costs

💡 Tip: All of this data appears in your Reflight dashboard for analysis and debugging. You can filter, search, and visualize your AI operations in real-time.

Running the Example

# Make sure you have your API key set (as environment variable)
REFLIGHT_API_KEY="your-reflight-key"

Simulation Setup

Reflight provides a simulation feature that allows you to test your Mastra agents with predefined datasets. This is useful for evaluating agent behavior, testing edge cases, and validating agent responses.

Step 1: Create a Simulation Dataset

Create your simulation dataset directly in the Reflight app:

Navigate to your Reflight dashboard and go to the Datasets section
Create a new dataset or select an existing one
Add traces to your dataset: Click "Add to Dataset" on any trace in your Reflight dashboard. The app automatically detects:
- The agent name
- The input messages
- The expected behavior (from the trace context)
Export the dataset: Once you've added all the traces you want to test, export the dataset as JSON

💡 Tip: You can add multiple traces to build a comprehensive test suite. The Reflight app automatically extracts all the necessary information from each trace, so you don't need to manually format the data.

The exported JSON will have the following format (example):

[
  {
    "agentName": "publisherAgent",
    "input": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Create a blog post about sustainable energy solutions"
          }
        ]
      }
    ],
    "expectedBehavior": "should orchestrate copywriting and editing to produce a final edited blog post about sustainable energy"
  },
  {
    "agentName": "copywriterAgent",
    "input": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Create a blog post about the benefits of TypeScript"
          }
        ]
      }
    ],
    "expectedBehavior": "should write engaging blog post copy about TypeScript benefits"
  },
  {
    "agentName": "editorAgent",
    "input": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Edit the following blog post only returning the edited copy: [blog post content]"
          }
        ]
      }
    ],
    "expectedBehavior": "should edit and improve the provided blog post copy"
  }
]

Save the exported JSON file to your project (e.g., data/sim-dataset.json).

Step 2: Run the Simulation

You can run simulations using either a JSON file or a TypeScript object directly. Create a script using MastraSimulator:

Option A: Using a JSON File

import { MastraSimulator } from "@reflight/mastra";
import { mastra } from "../mastra";
import fs from "fs/promises";
 
const simulator = new MastraSimulator(mastra);
 
const dataset = await fs.readFile("data/sim-dataset.json", "utf8");
const parsedDataset = JSON.parse(dataset);
 
const result = await simulator.runSimulation(parsedDataset);
 
console.log(result);

Option B: Using a TypeScript Object

You can also define your dataset directly as a TypeScript object:

import { MastraSimulator } from "@reflight/mastra";
import { mastra } from "../mastra";
 
const simulator = new MastraSimulator(mastra);
 
// Example: Testing the publisher agent workflow
const dataset = [
  {
    agentName: "publisherAgent",
    input: [
      {
        role: "user",
        content: [
          {
            type: "text",
            text: "Create a blog post about AI agents and their applications"
          }
        ]
      }
    ],
    expectedBehavior: "should orchestrate copywriting and editing by calling copywriterAgent and editorAgent in that order to produce a final edited blog post about AI agents"
  }
];
 
const result = await simulator.runSimulation(dataset);
 
console.log(result);

You can also test individual agents:

// Example: Testing the copywriter agent directly
const copywriterDataset = [
  {
    agentName: "copywriterAgent",
    input: [
      {
        role: "user",
        content: [
          {
            type: "text",
            text: "Create a blog post about microservices architecture"
          }
        ]
      }
    ],
    expectedBehavior: "should write engaging blog post copy about microservices"
  }
];
 
// Example: Testing the editor agent directly
const editorDataset = [
  {
    agentName: "editorAgent",
    input: [
      {
        role: "user",
        content: [
          {
            type: "text",
            text: "Edit the following blog post only returning the edited copy: TypeScript is a programming language..."
          }
        ]
      }
    ],
    expectedBehavior: "should edit and improve the provided blog post copy"
  }
];

Step 3: Execute the Simulation Script

Run your simulation script:

# Make sure you have your API key set
REFLIGHT_API_KEY="your-reflight-key"
 
# Run the simulation
tsx src/scripts/use-reflight-sim.ts

The simulator will execute each test case in your dataset and return results that you can analyze in your Reflight dashboard.

💡 Tip: Use simulations to validate agent behavior across different scenarios, test edge cases, and ensure consistent performance before deploying to production.

Langgraph