Welcome to PerOXO — an actor-based, real-time messaging backend written in Rust, designed to handle high-concurrency WebSocket communication with strong guarantees around performance, safety, and scalability.

PerOXO is not a monolith. It is a microservices-driven system built around Tokio’s async runtime, Axum WebSockets, gRPC, and message-passing concurrency, eliminating shared mutable state entirely.

At its core, PerOXO enables direct messaging and room-based chats, with durable persistence, asynchronous processing, and production-grade observability.

What Makes PerOXO Different?

Most real-time chat systems rely on locks, shared state, or heavy frameworks.
PerOXO instead embraces:

• Pure actor-model architecture
• Lock-free concurrency
• Message-passing only (no shared mutable state)
• Separation of concerns via microservices
• Observability-first design
• Each logical unit runs in its own async task, owns its own state, and communicates only through Tokio channels.

Core Architecture

PerOXO is designed using the Actor Model, structured around three primary actors. Each actor isolates state and handles specific responsibilities to ensure high concurrency and fault tolerance.

1.MessageRouter Actor

Role: The Central Hub

The MessageRouter acts as the singleton orchestrator for the application. It manages the global state of the chat server and routes messages between sessions.

• State Management:
  • Active Users: Maintains a registry of currently connected users.
  • Presence: Tracks online/offline status (PresenceStatus).
  • Room Mappings: Manages which users are subscribed to which chat rooms.
• Key Responsibilities:
  • Lifecycle Management: Handles User Registration (Connect) and Deregistration (Disconnect).
  • Routing:
    • Direct Messages: Routes packets from Sender → Receiver UserSession.
    • Room Messages: Broadcasts packets to all UserSession actors in a specific room.
  • Query Handling: Responds to requests regarding user presence or room participants.

2.UserSession Actor

Role: Connection Handler
Scope: One Actor per WebSocket connection

The UserSession represents a specific client connection. It wraps the raw WebSocket stream and bridges the gap between the client and the MessageRouter.

• Ownership:
  • Identity: Holds the TenantUserId (Project ID + User ID).
  • Transport: Owns the WebSocket Stream (Split into Sink/Stream).
  • Channels: Manages inbound (Client → Server) and outbound (Server → Client) MPSC channels.
• Async Event Loops:
  1. Read Loop (Receiver): Listens for incoming WebSocket frames, deserializes them, and forwards them to the MessageRouter.
  2. Write Loop (Sender): Listens for internal messages (from Router or System) and serializes them into WebSocket frames to send to the client.
• Lifecycle: Automatically cleans up resources and notifies the Router upon disconnection.

3.PersistenceActor

Role: Data Reliability & Storage Integration: gRPC $\rightarrow$ chat-service $\rightarrow$ ScyllaDB

The PersistenceActor decouples the real-time message flow from database operations. This ensures that database latency never blocks the WebSocket loop.

• Key Responsibilities:
  • Asynchronous Writes: Buffers messages intended for long-term storage (ScyllaDB).
  • Service Communication: Establishes and manages the gRPC connection to the chat-service.
  • Reliability Logic:
    • Retry Mechanism: Implements exponential backoff for failed writes.
    • Failure Isolation: Ensures that if the DB is down, live message delivery (RAM-to-RAM) continues uninterrupted.
• Philosophy: "Fire and Forget" for the hot path, but "Eventually Consistent" for storage.

Microservices Ecosystem

End-to-End Message Flow

The following sequence diagram illustrates how a single WebSocket message flows through the system — from client input, through routing, broadcasting, persistence, and back to connected users.

Multi-Tenancy Model & Identity Design

PerOXO implements multi-tenancy through a layered authentication and authorization system that ensures strict tenant isolation at the protocol level.

##1.Tenant Isolation Strategy

PerOXO uses a dual-identifier approach to guarantee isolation between different organizations or projects hosting on the platform.

• Tenant Identity: Defined by a project_id (format: peroxo_pj_{nanoid}) and a secret_api_key.
• User Identity: Users are identified not just by their ID, but by the combination of their project_id and user_id.

This structure ensures that a user ID 123 in Project A is mathematically distinct from user ID 123 in Project B.

##2.How Tokens Encode Tenant Context Identity is not inferred; it is explicitly encoded into the authentication token itself.

• Token Format: Tokens are opaque strings with a pxtok_ prefix followed by 24 alphanumeric characters.
• Payload (Redis): The token resolves to a JSON payload in Redis that carries the full tenant context:

pub struct UserToken {
    pub project_id: String,  // Tenant identifier
    pub user_id: String,     // User within tenant
    pub expires_at: u64,     // Time-limited access
}

This ensures that every authenticated session carries an immutable tenant binding.

3. Preventing Cross-Tenant Message Leakage

Cross-tenant access is prevented through multiple validation checkpoints before a user can ever send or receive a message:

• Token Generation Guard: The Auth Service validates the project_id against the secret_api_key stored in PostgreSQL before issuing a token. If the keys do not match, no token is created.
• WebSocket Handshake Gate: Before a WebSocket connection is upgraded, the handler extracts the token and calls the Auth Service to verify it.
• Strict Verification: The system verifies that the token exists, is valid, and has not expired. Only then is the UserToken object (containing the project_id) returned to the gateway.

4. Router-Level Enforcement

Multi-tenancy is enforced before the traffic reaches the routing logic.

• The MessageRouter operates on sessions that have already been strictly authenticated.
• By the time a RegisterUser message reaches the router, the system has already guaranteed that the connection belongs to a valid tenant.
• This separation of concerns allows the Router to focus on high-performance message passing without performing redundant database lookups for every packet.

5. Why This Matters

This design positions PerOXO as enterprise-grade collaboration infrastructure rather than a simple chat server:

• Enterprise Isolation: Each tenant operates in a secure silo with cryptographic separation.
• Scalable Multi-Tenancy: A single service instance can serve multiple organizations safely.
• Audit Trail: All actions are traceable to specific project_id + user_id combinations.
• Security: Short-lived tokens (600 seconds) minimize credential exposure window.