Hytale Multiserver Architecture Guide

Learn how to build sophisticated Hytale server networks with player referral systems, connection redirects, fallback servers, and custom QUIC proxies. Essential reading for server providers and network administrators.

Quick Navigation

Architecture Overview

Hytale's multiserver architecture allows you to build complex server networks that can:

  • Transfer players between servers without disconnecting
  • Implement lobby systems and game mode hubs
  • Balance load across multiple game servers
  • Provide fallback servers when primary servers are full
  • Build minigame networks with seamless transitions
  • Create proxy layers for DDoS protection

Core Concepts

Player Referral

API to send players to different servers while maintaining their session and authentication state.

Connection Redirect

Handshake-level redirection before player fully connects, useful for maintenance mode or server selection.

Fallback Server

Automatically send disconnected players to alternative servers instead of kicking them.

QUIC Proxy

Custom proxy server built with Netty QUIC to route traffic between players and backend servers.

Example Network Topology

┌─────────────┐
│   Players   │
└──────┬──────┘
       │
       ▼
┌─────────────────┐
│  Proxy Server   │ ← Single entry point (port 5520)
│  (QUIC Proxy)   │
└────────┬────────┘
         │
    ┌────┴────────────┬──────────┐
    ▼                 ▼          ▼
┌────────┐      ┌─────────┐  ┌──────────┐
│ Lobby  │      │ Minigame│  │ Survival │
│ Server │      │ Server  │  │  Server  │
└────────┘      └─────────┘  └──────────┘
    │                │             │
    └────────┬───────┴─────────────┘
             ▼
    [Player Referral API]

Players connect to a single proxy server which distributes them to appropriate backend servers based on your logic.

Player Referral System

The Player Referral API allows you to transfer connected players to different servers seamlessly.

Using PlayerRef.referToServer()

The core API for player transfers:

// Java example from HytaleServer.jar
import com.hytale.server.api.PlayerRef;
import com.hytale.server.api.ServerAddress;

// Transfer player to another server
public void sendPlayerToMinigame(PlayerRef player) {
    ServerAddress destination = new ServerAddress(
        "minigame.example.com",
        5520
    );

    // This transfers the player while maintaining their session
    player.referToServer(destination);

    // Player's client will seamlessly connect to new server
    // Authentication state is preserved
}

Session Continuity

When a player is referred to another server:

  • Authentication persists: Player doesn't need to re-authenticate
  • Seamless transition: No disconnect screen shown to player
  • Transfer token issued: Source server generates token for destination server
  • Timeout protection: Transfer must complete within 30 seconds or player disconnects

Practical Use Cases

Lobby → Game Server

Players spawn in a lobby server where they can select game modes. Clicking on a game mode portal transfers them to the appropriate game server.

// Player clicks on "Skywars" portal
player.referToServer(new ServerAddress("skywars1.network.com", 5520));

Game End → Lobby

When a minigame ends, automatically send players back to the lobby instead of leaving them in an empty arena.

// Game ended, send all players to lobby
for (PlayerRef player : game.getPlayers()) {
    player.referToServer(lobbyAddress);
}

Load Balancing

Dynamically distribute players across multiple game servers based on current load.

// Find least populated server
ServerAddress leastPopulated = findLeastPopulatedServer();
player.referToServer(leastPopulated);

Cross-Server Teleportation

Build a network where different worlds are hosted on separate servers, with portals that transfer players between them.

// Player enters nether portal
player.referToServer(netherServerAddress);

Error Handling

Handle transfer failures gracefully:

try {
    player.referToServer(destinationAddress);
    logger.info("Transferred player {} to {}",
        player.getUsername(), destinationAddress);
} catch (ServerUnavailableException e) {
    // Destination server is offline or unreachable
    player.sendMessage("§cThat server is currently offline.");
    // Optionally try fallback server
    player.referToServer(fallbackAddress);
} catch (TransferTimeoutException e) {
    // Transfer didn't complete in time
    player.sendMessage("§cTransfer timed out. Please try again.");
} catch (PlayerTransferException e) {
    // Generic transfer failure
    logger.error("Failed to transfer player", e);
    player.disconnect("Transfer failed");
}

Connection Redirects

Connection redirects occur during the handshake phase, before the player fully connects. This is useful for maintenance mode, version mismatches, or server selection.

Handshake-Level Redirect

Intercept connection during handshake and redirect to another server:

// Plugin/Mod example
@EventHandler
public void onPlayerHandshake(PlayerHandshakeEvent event) {
    if (serverMaintenanceMode) {
        // Redirect to maintenance lobby
        event.redirectTo(new ServerAddress("maintenance.network.com", 5520));
        event.setRedirectMessage("§eServer is in maintenance mode.");
    }

    if (event.getClientVersion() != REQUIRED_VERSION) {
        // Redirect outdated clients to update server
        event.redirectTo(updateServerAddress);
        event.setRedirectMessage("§cPlease update your client.");
    }
}

Maintenance Mode Implementation

Redirect all players to a maintenance lobby during updates:

Scenario:

You need to restart your main game servers for updates, but you don't want to kick all players offline.

Solution:

  1. Keep a lightweight "maintenance lobby" server running
  2. Enable maintenance mode on main servers, which redirects new connections to lobby
  3. Use player referral to move existing players to lobby
  4. Update and restart main servers
  5. Once updated, refer all players back from lobby to main servers

Version Routing

Route players to version-specific servers automatically:

@EventHandler
public void onHandshake(PlayerHandshakeEvent event) {
    String clientVersion = event.getClientVersion();

    // Route to appropriate server based on version
    switch (clientVersion) {
        case "1.0.0":
            event.redirectTo(new ServerAddress("v1.network.com", 5520));
            break;
        case "1.1.0":
            event.redirectTo(new ServerAddress("v1-1.network.com", 5520));
            break;
        case "2.0.0":
            event.redirectTo(new ServerAddress("v2.network.com", 5520));
            break;
        default:
            event.deny("§cUnsupported client version: " + clientVersion);
    }
}

Geographic Routing

Redirect players to geographically closer servers for better latency:

@EventHandler
public void onHandshake(PlayerHandshakeEvent event) {
    String playerIP = event.getPlayerIP();
    String region = geoIPLookup(playerIP); // US, EU, ASIA, etc.

    ServerAddress regionalServer = switch(region) {
        case "US" -> new ServerAddress("us.network.com", 5520);
        case "EU" -> new ServerAddress("eu.network.com", 5520);
        case "ASIA" -> new ServerAddress("asia.network.com", 5520);
        default -> new ServerAddress("global.network.com", 5520);
    };

    event.redirectTo(regionalServer);
}

Fallback Servers

Fallback servers catch players who would otherwise be disconnected, providing a better user experience during server issues.

Configuring Fallbacks

Set up fallback behavior in server configuration:

// server.properties or config.yaml
fallback-server-enabled: true
fallback-server-address: lobby.network.com
fallback-server-port: 5520

# Priority list of fallback servers
fallback-servers:
  - lobby1.network.com:5520
  - lobby2.network.com:5520
  - maintenance.network.com:5520

Disconnect Events

Handle disconnect events and route to fallback:

@EventHandler
public void onPlayerDisconnect(PlayerDisconnectEvent event) {
    if (event.getReason() == DisconnectReason.SERVER_SHUTDOWN) {
        // Server shutting down - send to fallback
        ServerAddress fallback = getFallbackServer();

        try {
            event.getPlayer().referToServer(fallback);
            event.setCancelled(true); // Prevent actual disconnect
        } catch (Exception e) {
            // Fallback failed, allow disconnect
            logger.error("Fallback failed for player", e);
        }
    }
}

Smart Fallback Selection

Choose fallback server intelligently based on current conditions:

private ServerAddress getFallbackServer() {
    List<ServerAddress> fallbackServers = config.getFallbackServers();

    for (ServerAddress server : fallbackServers) {
        // Check if server is online
        if (!isServerOnline(server)) continue;

        // Check if server has capacity
        int currentPlayers = getPlayerCount(server);
        int maxPlayers = getMaxPlayers(server);
        if (currentPlayers >= maxPlayers) continue;

        // Check server latency/responsiveness
        if (getServerPing(server) > 100) continue;

        // This server is suitable
        return server;
    }

    // No fallback available, return first in list
    return fallbackServers.get(0);
}

Common Fallback Scenarios

Server Restart

When restarting a game server, move all players to lobby instead of disconnecting them.

Server Crash

If a backend server crashes, players automatically get transferred to a stable fallback server.

Server Full

When a server reaches capacity, new players are sent to an alternative server or queue lobby.

Maintenance Window

During scheduled maintenance, all traffic automatically routes to maintenance lobby with status updates.

Building Custom QUIC Proxies

Build custom proxy servers using Netty QUIC to route Hytale traffic. This enables advanced networking features like DDoS protection, load balancing, and protocol inspection.

Why Build a Proxy?

  • Single entry point: Players only need one IP address, proxy routes to backends
  • DDoS protection: Proxy can absorb attacks and rate limit connections
  • Load balancing: Distribute players across multiple backend servers
  • Protocol inspection: Log, filter, or modify packets in transit
  • Zero-downtime updates: Update backend servers without changing player-facing IP
  • Authentication offloading: Proxy handles OAuth2, backend servers trust proxy

Dependencies

Add Netty QUIC to your Maven/Gradle project:

<!-- Maven -->
<dependency>
    <groupId>io.netty.incubator</groupId>
    <artifactId>netty-incubator-codec-quic</artifactId>
    <version>0.0.62.Final</version>
</dependency>

// Gradle
implementation 'io.netty.incubator:netty-incubator-codec-quic:0.0.62.Final'

Basic Proxy Implementation

Minimal QUIC proxy for Hytale:

import io.netty.bootstrap.Bootstrap;
import io.netty.channel.*;
import io.netty.incubator.codec.quic.*;

public class HytaleProxy {
    private final int listenPort = 5520;
    private final String backendHost = "backend.network.com";
    private final int backendPort = 5520;

    public void start() throws Exception {
        EventLoopGroup group = new NioEventLoopGroup();

        try {
            // Configure QUIC server
            QuicSslContext sslContext = QuicSslContextBuilder
                .forServer(privateKey, certificateChain)
                .applicationProtocols("hytale")
                .build();

            ChannelHandler codec = new QuicServerCodecBuilder()
                .sslContext(sslContext)
                .maxIdleTimeout(30_000, TimeUnit.MILLISECONDS)
                .initialMaxData(10_000_000)
                .initialMaxStreamDataBidirectionalLocal(1_000_000)
                .initialMaxStreamDataBidirectionalRemote(1_000_000)
                .initialMaxStreamsBidirectional(100)
                .tokenHandler(InsecureQuicTokenHandler.INSTANCE)
                .handler(new ChannelInitializer<QuicChannel>() {
                    @Override
                    protected void initChannel(QuicChannel ch) {
                        ch.pipeline().addLast(new ProxyHandler());
                    }
                })
                .streamHandler(new ChannelInitializer<QuicStreamChannel>() {
                    @Override
                    protected void initChannel(QuicStreamChannel ch) {
                        ch.pipeline().addLast(new StreamProxyHandler());
                    }
                })
                .build();

            // Bind and start accepting connections
            Bootstrap bootstrap = new Bootstrap()
                .group(group)
                .channel(NioDatagramChannel.class)
                .handler(codec);

            Channel channel = bootstrap.bind(listenPort).sync().channel();
            System.out.println("Hytale Proxy listening on port " + listenPort);

            channel.closeFuture().sync();
        } finally {
            group.shutdownGracefully();
        }
    }
}

Packet Forwarding Handler

Forward packets between client and backend server:

public class StreamProxyHandler extends ChannelInboundHandlerAdapter {
    private Channel backendChannel;

    @Override
    public void channelActive(ChannelHandlerContext ctx) {
        // Establish connection to backend server
        Bootstrap backendBootstrap = new Bootstrap()
            .group(ctx.channel().eventLoop())
            .channel(NioDatagramChannel.class)
            .handler(new BackendConnectionHandler(ctx.channel()));

        backendBootstrap.connect(backendHost, backendPort)
            .addListener((ChannelFutureListener) future -> {
                if (future.isSuccess()) {
                    backendChannel = future.channel();
                } else {
                    ctx.close();
                }
            });
    }

    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) {
        // Forward client packets to backend
        if (backendChannel != null && backendChannel.isActive()) {
            backendChannel.writeAndFlush(msg);
        } else {
            ReferenceCountUtil.release(msg);
        }
    }

    @Override
    public void channelInactive(ChannelHandlerContext ctx) {
        if (backendChannel != null) {
            backendChannel.close();
        }
    }
}

public class BackendConnectionHandler extends ChannelInboundHandlerAdapter {
    private final Channel clientChannel;

    public BackendConnectionHandler(Channel clientChannel) {
        this.clientChannel = clientChannel;
    }

    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) {
        // Forward backend packets to client
        if (clientChannel.isActive()) {
            clientChannel.writeAndFlush(msg);
        } else {
            ReferenceCountUtil.release(msg);
        }
    }
}

Advanced Features

Connection Rate Limiting

private LoadingCache<String, AtomicInteger> connectionCounts =
    CacheBuilder.newBuilder()
        .expireAfterWrite(1, TimeUnit.MINUTES)
        .build(new CacheLoader<String, AtomicInteger>() {
            public AtomicInteger load(String key) {
                return new AtomicInteger(0);
            }
        });

@Override
public void channelActive(ChannelHandlerContext ctx) {
    String clientIP = getClientIP(ctx);
    int connections = connectionCounts.get(clientIP).incrementAndGet();

    if (connections > 5) {
        // Rate limit exceeded
        ctx.close();
        return;
    }

    super.channelActive(ctx);
}

Packet Inspection & Logging

@Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
    ByteBuf packet = (ByteBuf) msg;

    // Log packet metadata
    logger.debug("Packet from {}: {} bytes",
        ctx.channel().remoteAddress(),
        packet.readableBytes());

    // Inspect packet contents (be careful with sensitive data!)
    if (shouldInspect(packet)) {
        byte[] data = new byte[packet.readableBytes()];
        packet.getBytes(0, data);
        analyzePacket(data);
    }

    // Forward packet
    backendChannel.writeAndFlush(msg);
}

Dynamic Backend Selection

@Override
public void channelActive(ChannelHandlerContext ctx) {
    // Select backend based on load balancing algorithm
    ServerAddress backend = loadBalancer.selectServer();

    // Log selection
    logger.info("Routing player to {}", backend);

    // Connect to selected backend
    connectToBackend(ctx, backend);
}

Packet Definitions

Hytale's packet structure can be found in the HytaleServer.jar. Common packets for proxy implementation:

  • Handshake Packet: Initial connection, contains protocol version and player intent
  • Login Start: Player username and authentication token
  • Login Success: Server confirms successful authentication
  • Disconnect: Connection termination with reason
  • Keep Alive: Periodic ping to maintain connection

Decompile HytaleServer.jar with tools like JD-GUI or Jadx to inspect full packet definitions.

Load Balancing Strategies

Distribute players across multiple servers to optimize performance and resource utilization.

Round Robin

Simplest algorithm - rotate through servers sequentially:

public class RoundRobinBalancer {
    private List<ServerAddress> servers;
    private AtomicInteger currentIndex = new AtomicInteger(0);

    public ServerAddress selectServer() {
        int index = currentIndex.getAndIncrement() % servers.size();
        return servers.get(index);
    }
}

Pros: Simple, fair distribution. Cons: Doesn't account for server load or capacity.

Least Connections

Send players to server with fewest current connections:

public class LeastConnectionsBalancer {
    private Map<ServerAddress, Integer> connectionCounts;

    public ServerAddress selectServer() {
        return connectionCounts.entrySet().stream()
            .min(Map.Entry.comparingByValue())
            .map(Map.Entry::getKey)
            .orElse(fallbackServer);
    }

    public void onPlayerConnect(ServerAddress server) {
        connectionCounts.merge(server, 1, Integer::sum);
    }

    public void onPlayerDisconnect(ServerAddress server) {
        connectionCounts.merge(server, -1, Integer::sum);
    }
}

Pros: Balances load based on actual usage. Cons: Requires tracking connection counts.

Weighted Distribution

Assign weights to servers based on their capacity:

public class WeightedBalancer {
    private List<WeightedServer> servers;

    static class WeightedServer {
        ServerAddress address;
        int weight; // Higher weight = more traffic
        int currentConnections;
    }

    public ServerAddress selectServer() {
        return servers.stream()
            .max(Comparator.comparingDouble(s ->
                (double) s.weight / (s.currentConnections + 1)
            ))
            .map(s -> s.address)
            .orElse(fallbackServer);
    }
}

Example: High-spec server gets weight 10, low-spec gets weight 3. High-spec receives ~3x more traffic.

Geographic Routing

Route players to nearest server for best latency:

public class GeographicBalancer {
    private Map<String, List<ServerAddress>> regionServers;

    public ServerAddress selectServer(String playerIP) {
        String region = geoIP.lookup(playerIP); // MaxMind GeoIP
        List<ServerAddress> regionalServers = regionServers.get(region);

        if (regionalServers == null || regionalServers.isEmpty()) {
            // Fall back to global servers
            regionalServers = regionServers.get("GLOBAL");
        }

        // Use least connections within region
        return selectLeastConnected(regionalServers);
    }
}

Resource-Based Balancing

Select server based on real-time resource availability:

public class ResourceBalancer {
    private Map<ServerAddress, ServerMetrics> metrics;

    static class ServerMetrics {
        double cpuUsage;    // 0.0 - 1.0
        double memoryUsage; // 0.0 - 1.0
        int playerCount;
        long lastUpdate;
    }

    public ServerAddress selectServer() {
        return metrics.entrySet().stream()
            .filter(e -> System.currentTimeMillis() - e.getValue().lastUpdate < 30000)
            .filter(e -> e.getValue().cpuUsage < 0.8)
            .filter(e -> e.getValue().memoryUsage < 0.9)
            .min(Comparator.comparingInt(e -> e.getValue().playerCount))
            .map(Map.Entry::getKey)
            .orElse(fallbackServer);
    }
}

Requires monitoring infrastructure (e.g., Prometheus) to collect server metrics.

Security Considerations

DDoS Mitigation

  • Implement connection rate limiting per IP address
  • Use SYN cookies to prevent SYN flood attacks
  • Deploy proxy behind DDoS protection service (Cloudflare, etc.)
  • Implement IP blacklist/whitelist functionality
  • Monitor for unusual traffic patterns

Authentication Security

When building proxies that handle authentication:

⚠️ Important

  • Never log or store player authentication tokens
  • Use TLS/SSL for all inter-server communication
  • Validate tokens on proxy, don't pass raw tokens to backend
  • Implement token refresh logic properly
  • Use secure random for generating internal session IDs

Backend Server Trust

Secure communication between proxy and backend servers:

  • Use internal network/VPN for proxy-to-backend traffic
  • Implement shared secret authentication between proxy and backends
  • Whitelist proxy IP addresses on backend firewalls
  • Use mTLS (mutual TLS) for authentication if possible

Packet Filtering

Inspect and filter malicious packets at proxy level:

@Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
    ByteBuf packet = (ByteBuf) msg;

    // Check packet size
    if (packet.readableBytes() > MAX_PACKET_SIZE) {
        logger.warn("Oversized packet from {}", ctx.channel().remoteAddress());
        ReferenceCountUtil.release(msg);
        ctx.close();
        return;
    }

    // Check for malformed packets
    if (!isValidPacket(packet)) {
        logger.warn("Invalid packet from {}", ctx.channel().remoteAddress());
        ReferenceCountUtil.release(msg);
        return;
    }

    // Forward valid packet
    backendChannel.writeAndFlush(msg);
}

Monitoring & Observability

Track the health and performance of your multiserver network.

Key Metrics to Monitor

Connection Metrics

  • Active connections per server
  • Connection attempts per second
  • Failed connection rate
  • Average connection duration

Transfer Metrics

  • Player transfers per minute
  • Failed transfer rate
  • Average transfer time
  • Transfer success rate by destination

Server Health

  • Backend server availability
  • Server response time
  • CPU and memory usage
  • Network bandwidth utilization

Error Tracking

  • Connection timeouts
  • Authentication failures
  • Packet forwarding errors
  • Backend unavailable events

Prometheus Integration

Export metrics to Prometheus for visualization in Grafana. See our Recommended Plugins guide for the Prometheus exporter plugin.

// Example Prometheus metrics in proxy
private final Counter connectionsTotal = Counter.build()
    .name("hytale_proxy_connections_total")
    .help("Total connections")
    .register();

private final Gauge activeConnections = Gauge.build()
    .name("hytale_proxy_active_connections")
    .help("Active connections")
    .labelNames("backend")
    .register();

private final Histogram transferDuration = Histogram.build()
    .name("hytale_proxy_transfer_duration_seconds")
    .help("Player transfer duration")
    .register();

Logging Best Practices

  • Log levels: Use DEBUG for packet details, INFO for connections, WARN for issues, ERROR for failures
  • Structured logging: Use JSON format for easy parsing and analysis
  • Log aggregation: Send logs to centralized system (ELK stack, Loki, etc.)
  • PII protection: Never log authentication tokens or sensitive player data
  • Log rotation: Implement automatic log rotation to prevent disk fill

Health Checks

Implement health check endpoints for monitoring:

// HTTP health check endpoint
@GetMapping("/health")
public HealthResponse getHealth() {
    boolean allBackendsHealthy = backendServers.stream()
        .allMatch(this::isServerHealthy);

    return new HealthResponse(
        allBackendsHealthy ? "healthy" : "degraded",
        getActiveConnections(),
        getBackendStatuses()
    );
}

// Liveness check (is proxy running?)
@GetMapping("/health/live")
public String liveness() {
    return "ok";
}

// Readiness check (can proxy accept connections?)
@GetMapping("/health/ready")
public String readiness() {
    boolean ready = hasHealthyBackends() && !isOverloaded();
    if (!ready) {
        throw new ServiceUnavailableException();
    }
    return "ok";
}

Frequently Asked Questions

Can I use existing proxy software like BungeeCord or Velocity?

No. BungeeCord and Velocity are designed for Minecraft which uses TCP-based protocol. Hytale uses QUIC (UDP-based), requiring custom proxy implementations with Netty QUIC.

Do backend servers need to be modified to work with a proxy?

It depends on your architecture. For transparent proxying (just packet forwarding), no modifications needed. For advanced features like authentication offloading or shared state, you'll need to implement communication between proxy and backends.

What's the performance overhead of using a proxy?

Minimal if implemented correctly. Expect 1-5ms additional latency for packet forwarding. A well-optimized proxy on modern hardware can handle thousands of concurrent connections without bottleneck.

Can players see when they're transferred between servers?

It depends on the implementation. Using PlayerRef.referToServer() provides a seamless transition with no disconnect screen. However, there will be a brief loading screen as the new world loads.

The transition is similar to changing dimensions in vanilla Hytale.

How do I share player data across servers in a network?

Common approaches:

  • Shared database (MySQL, PostgreSQL) for persistent data
  • Redis for real-time data and caching
  • Message queue (RabbitMQ, Kafka) for events
  • Custom API endpoints on each server

What happens if a backend server crashes during player transfer?

The player will experience a connection timeout and be disconnected (or sent to fallback server if configured). Implement health checks before initiating transfers to prevent this.

Is it possible to have cross-server chat?

Yes, but it requires custom implementation. Use a message broker (Redis Pub/Sub, RabbitMQ) or direct server-to-server communication to relay chat messages between servers in your network.

Where can I find example multiserver implementations?

Check the official Hytale documentation and community forums. Large server providers (Nitrado, Apex Hosting) may also publish reference implementations or libraries for network management.

Related Guides

Server Manual

Learn the basics of running a Hytale dedicated server

Server Authentication

OAuth2 authentication for game server providers

Recommended Plugins

Essential plugins including WebServer and monitoring tools

Connection Troubleshooting

Fix network and connectivity issues

Need Managed Server Network?

Skip the complexity of building and managing multiserver infrastructure. Our enterprise hosting plans include managed proxy, load balancing, and 24/7 monitoring.

View Enterprise Plans →