# Introduction Nodeoperator AI is an autonomous node operator agent that deploys, manages, and remediates issues with blockchain infrastructure using GitOps as a human-in-the-loop control model. ## Why an AI Node Operator? Running blockchain infrastructure today is manual, fragile, and error-prone. Nodeoperators and Solo Stakers: * Manually Track upstream client releases * Perform risky upgrades * Debug failing nodes under time pressure * Maintain complex Kubernetes environments Nodeoperator AI is designed to reduce this operational burden while keeping humans in control. ## Is AI agent safe for critical infrastructure? We recognize the concerns about placing critical infrastructure under the control of an AI agent: "Is AI ready? Can it be trusted with critical infra? What about hallucinations and unpredictable execution?. Is this just jumping on another shining new AI tool?" These questions are valid and we addressed them head on Nodeoperator AI is built on a constraint-driven model, not open-ended automation: * Actions follow deterministic workflows * Operational boundaries are explicitly defined * Changes are delivered via GitOps (not direct mutations) * The agent uses domain-specific infrastructure knowledge * Human approval remains part of the control loop When sandboxed, scoped, and supervised, AI agents can reduce human error and execute repetitive operational tasks with higher consistency than manual workflows. ## System Architecture Nodeoperator AI is built as modular services: ### Interfaces Where operators interact with the system. Designed to fit existing workflows rather than forcing a chat-only model. #### Ponos **Ponos** is the command interface for Nodeoperator AI. > Ponos (Greek: Πόνος) means *toil*, *labor*, or *sustained effort*. Ponos takes on that toil for node operators. **Available today:** | Interface | Use Case | | --------- | -------------------------------------------------------------------------------------------------- | | **TUI** | Interactive terminal UI with workflow progress cards, real-time logs, and natural language input | | **Slack** | Chat interface for team workflows via natural language, slash commands, and threaded conversations | **TUI Features:** * Natural language command input * Session history and resume capability **Slack Features:** * Natural language chat interface * Slash commands for common operations * Thread-based conversations for follow-ups * Alert response integration * Team visibility into operations **Planned interfaces:** * **GitHub Actions** — Trigger workflows from CI/CD pipelines * **GitHub Comments** — Operate via PR/issue comments * **Discord** — Community and team workflows ### Agent Core (Backend) This is the agent's decision engine, where context logic, safety guardrails, and operational intelligence are enforced. * **Workflow orchestration** — Manages multi-step operations with checkpoints and rollback capability * **Session management** — Maintains conversation context and execution state across interactions * **LLM integration** — Supports Claude and GPT-4 with streaming responses * **Safety guardrails** — Validates actions against operational rules before execution * **Rulebook engine** — Applies team-defined playbooks and constraints to agent decisions * **Memory system** — Stores and retrieves operational knowledge for context-aware responses ### MCP Servers MCP (Model Context Protocol) servers are modular connectors to external systems. They are separated to allow teams to run their own servers, control credentials, and minimize trust assumptions. | Server | Purpose | | ------------------ | ---------------------------------------------------------------- | | **GitHub MCP** | Create PRs, manage issues, fetch releases, repository operations | | **Kubernetes MCP** | Query pods, fetch logs, read deployments, cluster operations | | **Slack MCP** | Read/send messages, manage threads, chat interface integration | | **Telescope MCP** | Privacy-preserving observability for blockchain infrastructure | | **Blockchain MCP** | Protocol-specific tooling for chain interactions | **Key design principles:** * **Self-hostable** — Run MCP servers in your own environment * **Credential isolation** — Each server manages its own secrets * **Minimal trust** — The agent only has access to what you explicitly connect * **Auditable** — All MCP calls are logged All MCP servers are open source: ## Core Workflows & Capabilities Ponos supports three core workflows: ### 1. Upgrade Workflow Upgrade blockchain clients and infrastructure components with automated changelog analysis. * **Supported clients**: Ethereum execution/consensus clients (Geth, Prysm, Lighthouse, Teku, Nimbus), EVM chains, Polkadot, Cosmos, and Solana (experimental) * **What it does**: * Fetches latest releases from upstream repositories * Analyzes changelogs and identifies breaking changes * Compares current vs target versions * Generates upgrade PR with AI-summarized release notes **Example prompts:** * "Upgrade mainnet Geth to the latest version" * "Show me available Lighthouse versions for testnet" * "Upgrade all Ethereum clients on holesky to latest stable" ### 2. Diagnose Workflow Investigate node failures using logs, metrics, and cluster state to determine root causes. * **What it does**: * Collects pod logs and Kubernetes events * Queries Prometheus/Grafana metrics * Performs root cause analysis (RCA) * Creates GitHub issues with findings * Generates fix PRs for common issues (e.g., memory limits, config errors) **Example prompts:** * "Diagnose mainnet Ethereum validators" * "Check why Geth pods are failing on testnet" * "Investigate high attestation miss rate on validator-01" ## Features ### GitOps-First Operations **Upgrade Nodes Through Pull Requests** Client upgrades are proposed via GitOps with version and release awareness. Every upgrade includes AI-generated changelog summaries, breaking change detection, and rollback instructions. **Operate Through Git, Not Direct Access** Infrastructure is never mutated directly — all changes go through reviewable PRs. This provides a complete audit trail, enables team review, and allows easy rollbacks. ### Intelligent Diagnostics **Root Cause Analysis (RCA)** When nodes fail, the agent correlates logs, metrics, and Kubernetes state to identify the root cause. Findings are documented in GitHub issues with actionable recommendations. **Automated Fix Generation** For common issues (OOM kills, resource limits, configuration errors), the agent generates fix PRs automatically. Human approval is still required before changes are applied. ### AI Capabilities **Natural Language Interface** Describe what you want in plain English. The agent interprets your intent and executes the appropriate workflow. **Context-Aware Sessions** The agent remembers conversation context. Follow-up questions like "now do the same for testnet" work without repeating the full context. **Multi-Model Support** Works with Claude and GPT-4. Choose the model that fits your needs and budget. ### Operational Safety **Keep Secrets Out of Outputs** Sensitive values (API keys, passwords, private keys) are automatically redacted and never exposed in logs, PRs, or agent responses. **Enforce Operational Guardrails** Actions are validated against safety rules before execution. The agent cannot perform destructive operations without explicit approval. **Rulebooks** Define operational playbooks that the agent must follow. Rulebooks encode your team's best practices and constraints. ### Observability & Tracking **Real-Time Progress** Workflows display live progress in the TUI. See exactly what the agent is doing at each step. **Execution History** All sessions are logged with checkpoints. Resume failed workflows or replay past operations. **Session Continuity** If a workflow fails, you can resume from the last checkpoint instead of starting over. ### Integration & Extensibility **Work From Your Existing Tools** Run operations from the Ponos terminal interface, Slack, or automation workflows (GitHub Actions coming soon). **Integrate With Your Stack** Connects to GitHub, Kubernetes, Prometheus, Grafana, and blockchain networks via MCP servers. **Run It in Your Own Environment** MCP servers are open source and self-hostable. You control credentials, network access, and trust boundaries. ### Multi-Chain Support **Ethereum Ecosystem** Full support for execution clients (Geth, Nethermind, Besu, Erigon) and consensus clients (Prysm, Lighthouse, Teku, Nimbus, Lodestar). **Other Networks** Polkadot, Cosmos, and Solana support (experimental). The architecture is designed to be chain-agnostic. # Ponos vs Coding Agents # Install Ponos Install the Ponos TUI on macOS or Linux. Choose your install path: * [Linux Installation](/getting-started/install-ponos/linux) * [macOS Installation](/getting-started/install-ponos/macos) # Installation on Linux Install Ponos on Linux. ## Quick install (recommended) ```bash curl -sSL https://nodeoperator.ai/install.sh | sh ``` This installs the `ponos` binary. ## Build from source Prereqs: * Go (1.20+) * Git * Make ```bash git clone https://github.com/blockops-sh/ponos cd ponos make build-ponos ``` The binary is produced at `./bin/ponos`. ## Download the binary from GitHub Releases Pick the correct asset for your CPU: * `ponos_Linux_x86_64.tar.gz` (amd64) * `ponos_Linux_arm64.tar.gz` (arm64) Replace `vX.Y.Z` with a real tag from the GitHub releases page. Example for amd64: ```bash VERSION="vX.Y.Z" curl -L -o ponos.tar.gz \ "https://github.com/blockopsnetwork/ponos/releases/download/${VERSION}/ponos_Linux_x86_64.tar.gz" tar -xzf ponos.tar.gz sudo mv ponos /usr/local/bin/ponos ``` Verify the binary is on your PATH: ```bash which ponos ``` ## Run the TUI ```bash ponos ``` This is the Ponos TUI after launch. You can begin prompting the agent here
## Upgrade Ponos ```bash ponos upgrade ``` You can target a specific version: ```bash ponos upgrade --version vX.Y.Z ``` # Installation on macOS Install Ponos on macOS. ## Quick install (recommended) ```bash curl -sSL https://nodeoperator.ai/install.sh | sh ``` This installs the `ponos` binary. ## Build from source Prereqs: * Go (1.20+) * Git * Make ```bash git clone https://github.com/blockops-sh/ponos cd ponos make build-ponos ``` The binary is produced at `./bin/ponos`. ## Download the binary from GitHub Releases Pick the correct asset for your CPU: * `ponos_Darwin_x86_64.tar.gz` (Intel) * `ponos_Darwin_arm64.tar.gz` (Apple Silicon) Replace `vX.Y.Z` with a real tag from the GitHub releases page. Example for Apple Silicon: ```bash VERSION="vX.Y.Z" curl -L -o ponos.tar.gz \ "https://github.com/blockopsnetwork/ponos/releases/download/${VERSION}/ponos_Darwin_arm64.tar.gz" tar -xzf ponos.tar.gz sudo mv ponos /usr/local/bin/ponos ``` Verify the binary is on your PATH: ```bash which ponos ``` ## Run the TUI ```bash ponos ``` This is the Ponos TUI after launch. You can begin prompting the agent here
## Upgrade Ponos ```bash ponos upgrade ``` You can target a specific version: ```bash ponos upgrade --version vX.Y.Z ``` # Configure Ponos Create a ponos.yml and set required integrations. Ponos reads configuration from `ponos.yml`. By default it looks for the file in: 1. `./ponos.yml` (current directory) 2. `~/.config/ponos/ponos.yml` 3. `/etc/ponos/ponos.yml` You can override the path with `PONOS_CONFIG_PATH`. ## Get your API key Ponos uses an API key to authenticate with the NodeOperator backend. 1. Go to `https://platform.nodeoperator.ai/`
This is the NodeOperator login screen where you start the magic-link flow. 2. Provide your email and a Magic link will be sent to you
Enter your email and use the link in your inbox to sign in. 3. Create or copy your API key
Copy the API key shown here and keep it safe. You will paste it into `ponos.yml`. 4. Add it to your `ponos.yml` as `api_key` ## Create a config file Copy the example config: ```bash cp ponos.yml.example ponos.yml ``` ## Configuring Ponos for Client Updates and Management This section is required for upgrade workflows and GitOps PR creation. At minimum, set: * `api_endpoint` (NodeOperator API base URL) * `api_key` (from ) * GitHub auth (PAT or GitHub App) * Slack bot token + signing secret Minimal example: ```yaml version: 1 api_endpoint: "https://api.nodeoperator.ai" api_key: "nai_..." integrations: github: token: "ghp_xxx" # OR GitHub App: # app_id: "123456" # install_id: "654321" # pem_key: "/path/to/your.pem" # bot_name: "ponos-bot" slack: token: "xoxb-xxx" signing_key: "your_slack_signing_secret" verify_token: "optional" channel: "sre-tasks" ``` Once you save `ponos.yml`, restart Ponos. To verify the TUI is connected to the API, send a simple prompt like “hello.” You should see a response like this:
## Projects (Release Workflow) Projects map networks to repos and file paths that Ponos can update: ```yaml projects: - network: "ethereum" project_name: "mainnet" owner: "your-org" name: "infra-repo" branch: "main" paths: - "deployments/ethereum.yaml" ``` ## Node Observability & Incident Management Use this section if you want diagnostics, logs, and metrics-driven analysis. ### Telescope (automatic monitoring for any blockchain) Telescope is an all‑in‑one observability tool for blockchain nodes. It removes the need to hand‑maintain separate scrape configs for each component by setting up metrics and logs automatically based on the network you run. You can read more about the project here: If you already run Prometheus and Loki, point Ponos at your own endpoints here. Make sure your metrics and logs are labeled with `project_id` or `project_name` so Telescope can scope queries correctly. ```yaml integrations: telescope: project_id: "" project_name: "" prometheus_url: "" prometheus_username: "" prometheus_password: "" loki_url: "" loki_username: "" loki_password: "" ``` ### Diagnostics Diagnostics is disabled by default. It only runs when `diagnostics.enabled` is set to `true`. The provider can be `telescope` or `kubernetes`. ```yaml diagnostics: enabled: true provider: "telescope" github: owner: "your-org" repo: "infrastructure-issues" slack: channel: "sre-tasks" kubernetes: namespace: "default" resource_type: "deployment" monitoring: service: "your-service-name" log_tail: 500 eval_interval: 2 ``` ## Automatic client updates (optional) Ponos can listen for release webhooks and create upgrade PRs automatically. This is off by default and only runs when the server is started with the release listener enabled. In `ponos.yml`: ```yaml server: enable_release_listener: true ``` And set the env var before starting the server: ```bash export ENABLE_RELEASE_LISTENER=true ``` ## Next step Run the TUI: ```bash ponos ``` # Using Ponos (demo) Demo setup using the public Playground repo. This page shows a sample demo setup using the public repo to represent an ideal infrastructire-as-code repository and how nodeoperator agent interacts with it, so you can test deploy, upgrade and diagnostics workflows without wiring a private repository. Repository: Example manifest used in this demo: ## Upgrade demo Add this to your `ponos.yml` under `projects`: ```yaml projects: - network: ethereum project_name: ethereum-hoodi-fullnode owner: blockopsnetwork name: playground branch: main paths: - ethereum/ethereum-hoodi.yaml ``` Example prompts: * "Upgrade ethereum hoodi to the latest version" * "Show me available Geth versions for hoodi" ### Upgrade demo video {% embed url="" %} ## Diagnostics demo Diagnostics only run when enabled in `ponos.yml` and the provider is set to `telescope` or `kubernetes`. Example prompts: * "Diagnose mainnet Ethereum nodes" * "Check why Geth pods are failing on testnet" {% embed url="" %} ## Ponos.yml config (Diagnostic) This is the full `ponos.yml` used in the demo, with secrets redacted: ```yaml version: 1 # dev api_endpoint: "http://localhost:8001" api_key: "nai-REDACTED" # prod # api_endpoint: "https://api.nodeoperator.ai" # api_key: "nai-REDACTED" integrations: github: token: "ghp_REDACTED" slack: token: "xoxb-REDACTED" team_id: "T016BSZRXPF" signing_key: "REDACTED" telescope: project_id: "random-id" project_name: "rnadom-name" prometheus_url: "" prometheus_username: "prometheus_username" prometheus_password: "******" loki_url: "https://loki." loki_username: "loki_username" loki_password: "********" diagnostics: enabled: true provider: telescope github: owner: "blockopsnetwork" repo: "playground" slack: channel: "sre-tasks" monitoring: service: "lively-elisabetta" log_tail: 10 eval_interval: 2 server: port: "8080" enable_release_listener: false projects: - network: ethereum owner: blockopsnetwork project_name: ethereum-hoodi-fullnode name: playground branch: main paths: - ethereum/ethereum-hoodi.yaml ``` # Architecture # Supported Networks # Upgrade Workflow # Diagnostics Workflow # GitOps Model # Session Management # Slack # GitHub # Personal Access Token # GitHub App # Intelligent Node Observability # Prometheus # Loki # Telescope # Kubernetes # EVM Chains # Ethereum # Node Diagnostic Example # Optimism # Node Diagnostic Example # Base # Node Diagnostic Example # Polkadot # Polkadot # Node Diagnostic Example # Hyperbridge # Node Diagnostic Example # API Overview # Authentication # Endpoints Reference # MCP Servers # GitHub MCP # Slack MCP # Kubernetes MCP # Observability MCP # Configuration Reference # Environment Variables # Supported Clients # Security # Troubleshooting