Troubleshooting
Look up by symptom: check the diagnostic first, then act. Every section comes from a problem we actually hit during delivery.
Below are the blockers we have actually run into during delivery. Check here first before coming to us.
istioctl / kubectl cannot reach the cluster (microk8s)
microk8s does not keep its credentials in ~/.kube/config. Manually installed tools read the default path, so
they report a connection failure:
mkdir -p ~/.kube
sudo microk8s config > ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
microk8s kubectl is unaffected — it connects directly to the credentials inside the snap.
Confirm the Istio version actually running
When Istio is installed manually, istioctl version may "not see" it because the namespace or revision differs.
The most reliable way is to look at the image tags directly:
kubectl get pods -A -l app=istiod \
-o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'
When multiple versions coexist, also confirm which revision is used for default injection:
istioctl tag list
Login succeeds but you are sent back to the login page forever
The symptom is the address bar bouncing between the console and the identity service, which looks like "login did not succeed" even though the login already succeeded.
The cause is in the console's handling logic: for any GraphQL error from the gateway that starts with
Access Denied:, the frontend treats it as "not logged in" and starts login again; on the identity service side
the SSO session is still there, so the redirect is silent, and it goes round and round.
Check exactly one place: DevTools → Network → POST /graphql → errors[0].message in the Response.
| Error text | Meaning | What to do |
|---|---|---|
Access Denied: Authentication required |
The request has no Authorization header |
Check whether POST /api/auth/token returns 200; a 401 means the session cookie was not set or cannot be decoded, see multiple replicas must share the same value |
Access Denied: <jwt error> (invalid signature / invalid algorithm / malformed) |
The gateway's public key is not the current one from the identity service | Switch to the current signing public key and restart the gateway, see the tethys-gateway section |
Access Denied: Malformed token claims |
The token is missing id or roles |
The identity service's token structure does not match what the gateway expects |
Access Denied: ADMIN role required |
The account has no ADMIN role |
No loop; the UI shows a message and stops. Add the role to the account and log in again |
One more way to tell them apart: if the loop is intermittent, you get in sometimes and only loop after a few refreshes, it is more likely a key mismatch across web replicas; if it loops every single time, it is more likely a wrong gateway public key.
Browser reports a CORS error, or /graphql is 404 under the same domain
First compare two paths under the same domain: a REST path such as /api/v1/namespaces returns data while
/graphql is 404, and the preflight (OPTIONS) returns 204 but without Access-Control-Allow-Origin — this is
not a CORS configuration problem; the hostname is bound to the wrong microservice.
Check in this order:
- Confirm
gateway.<YOUR_DOMAIN>is bound only totethys-gateway. If the same hostname is also attached to another microservice in the console (for exampletethys-shipping), the later binding wins. - After removing the duplicate,
/livezshould return{"status":"ok"}. - Then check whether the gateway's
CORSincludes the console Origin (both protocol and port must be correct). - Restart the gateway after the change:
CORSis read only at process startup.
Do not use the
Serverheader in the response to tell which upstream it is — the edge nginx adds its ownServerto every response, and a healthy Next service has one too. The response body andx-envoy-upstream-service-timeare more reliable.
The live list does not refresh and the console reports WebSocket errors
WebSocket goes through a separate proxy configuration on the gateway; it is not the same path as GraphQL:
- The host in
SHIPPINGWS_ENDPOINTshould be Shipping's Service name, and you must not add abasePath—/watchand/xterminallive at Shipping's root path. WS_PROXIESmust list every watch path individually. Matching compares the whole pathname exactly; there is no prefix matching. Miss one and only that one connection breaks, while everything else works.- On the browser side,
wsEndpoint.BASEPATHmust equal the prefix of the gateway's public path (defaultshippingws), so the final address looks likewss://<CONSOLE_DOMAIN>/shippingws/watch/api/v1/namespaces.
After changing WS_PROXIES you also need to restart the gateway.
Bundle apply succeeds but there are no Pods
The operator is not up, or the CRDs are not installed. See "Install the PackingTable operator" in Base layer installation.
Troubleshooting quick reference
| Symptom | Where to look first |
|---|---|
Pod ImagePullBackOff |
Whether the image string matches {registry}/{app}:{version}; whether app equals the registry repository name; whether the tag exists; whether the target tenant has a pull Secret. You can confirm the image with docker manifest inspect <repo>:<tag> without exposing credentials |
Pod Waiting / never ready |
The Deployment exists but the container does not start: check events, image architecture (builds on Apple Silicon need --platform linux/amd64), required environment variables, and whether the mounted Secret exists |
Bundle phase: Failed |
status.message and conditions; most commonly the version identity (releaseVersion, or the vN inferred from the tag) is not greater than the current version, contains a dot, or the image does not exist |
Bundle stuck at Provisioning |
Whether the operator is running, whether the CRDs are installed, and whether the operator's ServiceAccount has enough permissions |
CR stuck at Terminating |
Whether the finalizer name matches the one registered with KubeOps (fewbox.com/<kind>finalizer) |
| Import returns 413 | Both layers need adjusting: the gateway's GRAPHQL_BODY_LIMIT and the edge nginx's client_max_body_size |
| Login succeeds but the gateway keeps treating you as not logged in | Whether IDP_SESSION_SECRET has the same value in all three of auth / gateway / payment; whether the JWT public key comes from the current signing key |
The page shows Access Denied: invalid signature |
The verification public key and auth's signing private key are not a pair, or the kid was rotated without updating the public key |
| The agent's MCP tools are unavailable | Whether *_ENDPOINT includes the /mcp path and points to the correct Service short name; whether the corresponding MCP Pod is healthy |
| Scheduled jobs do not run | Whether schedule is a valid cron expression; whether the CronJob is suspended due to an image / permission failure |