All Systems are Operational
Upcoming Maintenances

Updates

  1. RESOLVED

    All systems have been operating normal again.

  2. MONITORING

    What happened? Hytale deployed a new patch on the 24th which appears to change how world data is read/written to disk. This triggered a rare edge case in our ZFS filesystem, which we use to keep your data safe (always stored on two disks).

    We became aware of issues around 24 January 2026 at 20:00. Initially this looked hardware-related, so we migrated disks and checked affected nodes. At 25 January 2026 at 03:00 we migrated affected nodes to other systems. This finished at 25 January 2026 at 04:00, but the issue reappeared at 25 January 2026 at 10:30, now also affecting other Hytale nodes.

    Until 25 January 2026 at 16:00, service kept degrading again after ~45 minutes. At this point we ruled out hardware and identified Hytale servers as the root cause, so we disabled them to protect other customers.

    From 25 January 2026 at 17:00 onward we deployed emergency fixes across all nodes. This finished at 25 January 2026 at 20:45. Since then, we’ve seen no further Hytale-related degradation.

    We’re very sorry for the downtime and disruption. We've been working around the clock to get this fixed as soon as possible.

  3. IDENTIFIED

    We implemented a hotfix 3 hours ago. According to our monitoring everything seemed fine. But I/O delay increased in the last 30 minutes back to a value that causes performance issues.

    The issue seems to be related to a new version of Hytale that came out yesterday. We temporarily disabled Hytale servers on nodes mia and amy.

    We'll continue working on this and keep you updated.

    Thank you again for your patience and understanding!

  4. INVESTIGATING

    We are aware of ongoing issues with HypeServ node mia and amy.

    Yesterday, we believed we had a hardware failure on node amy. At 3am EST, we began migrating the entire node to a new hardware host, and by 4am EST all servers on node amy were back online. Everything ran smoothly from 4am to around 10am EST.

    However, starting around 12am, we began experiencing the same issue on node mia. At the moment, it appears that customer servers running Hytale may be triggering the problem, but we're still investigating the root cause.

    We’ll continue working on this and keep you updated. We will compensate you accordingly.