12. Troubleshooting

This chapter gathers the facts that admins need when troubleshooting problems or helping users troubleshoot theirs. Most of it points back to settings already explained in the previous chapter (§11). For details on voice quality, see §2.2.

12.1 Troubleshooting Voice Connections (TURN / Firewall / NAT)

When users behind strict NATs (VPN, corporate networks) can’t connect to voice, enabling the built-in TURN server can relay their connections. As the description on the TURN tab says: “Run a TURN relay server to help users behind strict NATs (VPN, corporate networks) connect to voice channels.” To enable it, see §11.1.

Firewalls and port forwarding are common sticking points. Make sure the UDP and TCP ports (default 3478) are both open in your firewall. If the server is behind a router, also forward these two ports to the server. TCP is the fallback for networks where UDP is blocked.

One key point to keep in mind while troubleshooting: Run Diagnostic is only an internal self-test of the server, and a successful diagnostic doesn’t mean that users can definitely connect. As the note below the button says: “This is an internal self-test only. It does NOT verify external connectivity. To confirm TURN works for real users, test with a VPN or from an external network.” So after the diagnostic passes, be sure to also test a voice connection over a VPN or from an external network. Only then can you confirm that TURN works for real users.

Also, if this server points to an external TURN through the TURN_URL environment variable, the built-in TURN is disabled, and a matching notice appears at the top of the TURN tab. To use the built-in TURN instead, you first need to remove TURN_URL.

Voice quality itself (for example, the trade-off between audio quality and bandwidth) is part of each channel’s voice settings. See §2.2 for details; this chapter doesn’t repeat it.

12.2 Troubleshooting Failed Server Updates

If an update fails automatic verification (SHA256), the file isn’t necessarily broken. It may only be a temporary network problem, or the hash list hasn’t been uploaded yet. In that case, you can go to the GitHub Release page and compare the hashes manually. If they match the hashes on that page, choose to skip verification and apply. For the steps, see §11.5.

A server that runs in a container (Docker) is updated by pulling the new version and recreating the container. Existing data is kept and isn’t lost. The command is:

docker compose pull && docker compose up -d

When an update fails, admins may see one of the following messages. The wording itself tells you where to look:

• “Insufficient disk space (need {needed}, {available} free).” — Free up disk space, then try again. An orphan files scan (§11.4) can also free up some space.
• “Download stalled — no data for 60 s. Please retry.” — Usually a network interruption. Retrying is enough.
• “Download exceeded the 30-minute timeout.” — This also points to a network problem. Retry once the network is stable.
• “Failed to check for updates.” and “Failed to download update.” — First check that the server’s outbound network connection is working, then retry. If a firewall restricts the server’s outbound connections, also confirm that the domains below are all allowed.

When it checks for and downloads updates, the server connects to HakoSpace’s software distribution source, and connects to GitHub instead when that source is unavailable (§11.5). If the server’s network allows connections only to specific domains, allow the server to connect over HTTPS (port 443) to the following domains:

• dl.hakospace.com
• api.github.com, github.com
• *.githubusercontent.com (downloads from GitHub are redirected to hosts under this domain)

Before an update is applied, the confirmation step warns: “This will disconnect all users and restart the server. Are you sure?” Do this during off-peak hours, when fewer users are online.

For a server that runs as a systemd service, if it never comes back after an update is applied, check that the service configuration uses Restart=always (see Quick Self-Host Guide §4.5). When applying an update, the server swaps in the new executable and then exits on its own, and systemd has to start the new version. With Restart=on-failure, systemd doesn’t restart it in this situation. After you fix the service configuration, reload the configuration and start the service (for example, sudo systemctl daemon-reload and sudo systemctl start hakospace). The new version that was already swapped in then starts running.

12.3 When Users Report Disconnections

When a user’s connection drops, their screen shows “Connection lost — reconnecting…”, along with “We’ll reconnect automatically once the connection is back — no action needed.” Once the connection is restored, it shows Reconnected.

An important point for admins: this message is deliberately worded neutrally, because a connection can fail for many reasons, including the user’s own network. So when a user sees “Connection lost — reconnecting…”, the server isn’t necessarily at fault. You can first ask the user to check their own network. In most cases, they reconnect automatically after a short wait.

“Unable to reach HakoSpace servers” (or “Could not reach HakoSpace servers. Please try again later.”) is a different matter: these messages appear only in the HakoSpace Improvement Program section of the Overview tab in the settings panel. They are about whether this server can reach HakoSpace, not about whether members can connect to this server; see §1.1.