Install Istio
The ingress gateway, in-cluster routing and sidecar injection. Installed right it is transparent; installed wrong it shows up as 503s and inexplicable 404s.
← Documentation index · Previous: Cluster preparation · Next: Configure PostgreSQL
FewBox uses Istio for exactly two things: giving external traffic an entry point, and routing to a specific service by hostname inside the cluster. Circuit breaking, rate limiting and advanced traffic management are not used — so install the minimal profile and do not introduce extra CRs.
1. Version: pin the minor version, do not use the addon
The reference environment measured 1.30.2. Istio's behavior differs across minor versions (especially how the Gateway / VirtualService fields
evolve, and the mTLS defaults), and it changes routes silently in places that never report an error, so:
- Pin one minor version (say 1.30.x) and do upgrades as a separate change;
- Do not use
microk8s enable istio— that addon's version follows the snap; it stays at whatever it was when the machine was provisioned, and upgrading the snap upgrades Istio along with it; - After installing, confirm the actual version from the image tag; do not trust
istioctl version(it may "not see" the installation because of a different revision):
kubectl get pods -A -l app=istiod \
-o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'
2. Install
Installing from the release archive is the most reliable path; when the target host cannot reach GitHub, download it locally and scp it over (the full mainland China treatment is in Mainland China network):
# Fetch the release archive on a machine with internet access: https://github.com/istio/istio/releases
scp istio-1.30.0-linux-amd64.tar.gz <user>@<host>:/root/
tar -xzf istio-1.30.0-linux-amd64.tar.gz
cd istio-1.30.0
export PATH=$PWD/bin:$PATH
istioctl version
Before installing, confirm the image can be pulled (replace the tag with the minor version you pinned; with image mirrors
in play, add --hosts-dir — see Mainland China network):
sudo microk8s ctr images pull docker.io/istio/pilot:1.30.0
Install:
istioctl install --set profile=default --set hub=docker.io/istio
--set hub=docker.io/istio replaces registry.istio.io in the official manifests with the official
images on Docker Hub. Both registries work; switching to Docker Hub is simply more convenient when you need image
mirrors (as on a mainland China network) — see Mainland China network.
Keep sidecar resource requests low
The default profile reserves 100m CPU / 128Mi memory for every istio-proxy. On a single 4C/8G machine, multiplied by the number of Pods, that
consumes the entire schedulable capacity outright (new Pods stay Pending). Start at 25m:
istioctl install --set profile=default --set hub=docker.io/istio \
--set values.global.proxy.resources.requests.cpu=25m
After installing, check what the node actually allocates:
kubectl describe node <node-name> | grep -A6 "Allocated resources"
3. Entry point: use NodePort when there is no LoadBalancer
Single-node MicroK8s has no MetalLB, so istio-ingressgateway is of type LoadBalancer but EXTERNAL-IP stays
<pending> forever. That does not stop you from using it — K8s still assigns it a NodePort:
kubectl -n istio-system get svc istio-ingressgateway
How the ports are split in the reference environment:
| Service port | NodePort | Purpose |
|---|---|---|
| 80 | <http-nodeport> |
All public HTTP, reverse-proxied in by the edge nginx |
| 443 | <https-nodeport> |
No TLS server is configured, not used in practice; TLS terminates at the edge |
| 15021 | <status-nodeport> |
Readiness probe |
Look up your own NodePort:
kubectl -n istio-system get svc istio-ingressgateway \
-o jsonpath='{.spec.ports[?(@.port==80)].nodePort}{"\n"}'
This value is the port to fill into proxy_pass in the edge nginx.
Do not use the NodePort on 443 directly. The port being open does not mean the handshake works — when the
Gatewayhas noTLSserver configured, a request to it is plain HTTP or gets disconnected. Put the certificate at the edge, see the next chapter.
4. The shape of the Gateway and VirtualServices
The entry point has two Gateway resources (both in the application namespace, both with selector pointing at istio: ingressgateway):
| Gateway | hosts | Port | Purpose |
|---|---|---|---|
internet |
The product domain and its wildcard (titanides.com, *.titanides.com) |
80 / HTTP | Public entry point |
intranet |
*.internal.titanides.com |
80 / HTTP | Direct intranet access, for debugging |
How to name internal domains: do not use
.local— RFC 6762 reserves it for mDNS, so macOS and any Linux running Avahi send a multicast query first, which makes resolution slow and occasionally fail outright..lanis common but has never been reserved: it could be delegated later, and it can collide with a public domain the customer already owns. In July 2024 ICANN reserved.internalfor private networks permanently (IANA announcement). The simplest option is a subdomain of a domain you own (such asinternal.titanides.com) with split-horizon DNS; when you genuinely need a standalone suffix, use.internal.
internet declares only port 80 and has no tls: section — that is deliberate: the certificates live outside the cluster,
and Istio only looks at the Host header. So when diagnosing "the domain opens to a 404", the first step is always to confirm that the request's
Host exactly matches the hosts of some VirtualService, not to go check certificates.
One VirtualService for every externally exposed service. The reference environment has 12, in three groups:
- Public: one each for
accounts.titanides.com,gateway.titanides.com,titanides.comandwww.titanides.com, and the console domain; - Intranet: admin consoles such as
tethys-*.titanides.com; - mesh: two
*-meshentries,gateways: ["mesh"], with the Service name inhosts(e.g.fewbox-thia-gateway). These two make Pods inside the cluster also go through Istio routing when they call each other by Service name — without them, sidecar-to-sidecar calls bypass the VirtualService, and CORS and timeout behavior differ from external access.
The easiest place to get wrong: routes point at a subset
The reference environment's VirtualServices do not connect to the Service directly, but point at a DestinationRule subset:
# VirtualService (fragment)
http:
- route:
- destination:
host: fewbox-thia-gateway
port: { number: 80 }
subset: v4 # <- points at the subset, not the Service
---
# DestinationRule (fragment)
spec:
host: fewbox-thia-gateway
subsets:
- name: v4
labels: { version: v4 } # <- the subset selects Pods by label
trafficPolicy:
tls: { mode: ISTIO_MUTUAL }
So upgrading a service means changing four places in sync; miss one and you get a 503 (no healthy upstream):
| # | Location | Reference environment value |
|---|---|---|
| 1 | Deployment name | fewbox-<service>-v4 |
| 2 | Pod template label version |
v4 |
| 3 | DestinationRule.subsets[].name and labels.version |
v4 |
| 4 | destination.subset in the VirtualService |
v4 |
When deploying with a Bundle the operator generates all four together and they cannot drift; when changing the image version by hand it is easy to change only place 1, and the symptom is brand-new Pods,
kubectl get podsall healthy, but the domain returns 503 as soon as you visit it. To confirm: the subset labels inkubectl -n <ns> get dr <service> -o yamldo not match the Pods' actual labels.
5. Sidecar injection and mTLS
Label the namespace to inject the whole namespace:
kubectl label namespace <namespace> istio-injection=enabled
After injection every Pod is 2/2 (application container + istio-proxy). Existing Pods do not get a sidecar added automatically; run
kubectl rollout restart deploy.
The reference environment's policy:
- There is no
PeerAuthenticationby default, so mTLS is permissive, not STRICT. This matters: clients without a sidecar (such as scheduled jobs inside the cluster) can still reach the services; - If you enable STRICT mTLS, the CronJob that pokes the agent every minute in scheduled jobs must be labeled
sidecar.istio.io/inject: "false"and must not go through STRICT, or the heartbeat stops silently; - The Service for a given service selects Pods with
app: fewbox-<service>; the sidecar fills in the canonical labels.
6. Observability (optional)
The reference environment does not enable tracing: istio-system contains only istiod and ingressgateway, no
jaeger / prometheus / kiali, and meshConfig has no enableTracing. Even if the application's
OTEL_EXPORTER_OTLP_ENDPOINT has an address filled in, no data is sent while TRACING_ENABLED=false.
When you need it, enable it in this order (the example installs the official addons; suitable for POC only):
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.30/samples/addons/prometheus.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.30/samples/addons/jaeger.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.30/samples/addons/kiali.yaml
Jaeger's Service only opens 14268 / 14250 / 9411 by default, without OTLP's 4317/4318. To receive OTLP you must flip the switch and add the ports:
kubectl -n istio-system set env deploy/jaeger COLLECTOR_OTLP_ENABLED=true
kubectl -n istio-system patch svc jaeger-collector --type=json -p='[
{"op":"add","path":"/spec/ports/-","value":{"name":"otlp-grpc","port":4317,"targetPort":4317,"protocol":"TCP"}},
{"op":"add","path":"/spec/ports/-","value":{"name":"otlp-http","port":4318,"targetPort":4318,"protocol":"TCP"}}
]'
Then register the provider and make the mesh use it by default:
kubectl -n istio-system patch configmap istio --type merge -p '
data:
mesh: |
enableTracing: true
defaultProviders:
tracing:
- otel-tracing
extensionProviders:
- name: otel-tracing
opentelemetry:
service: jaeger-collector.istio-system.svc.cluster.local
port: 4317
- name: zipkin
zipkin:
service: zipkin.istio-system.svc.cluster.local
port: 9411
'
kubectl -n istio-system rollout restart deploy/istiod
kubectl rollout restart deploy -n <app-namespace>
Control the sampling rate with a Telemetry resource (telemetry.istio.io/v1alpha1, still v1alpha1 on 1.30):
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: mesh-tracing
namespace: istio-system
spec:
tracing:
- providers:
- name: otel-tracing
randomSamplingPercentage: 100.0 # debug period; drop below 10 once stable
Verify that tracing really works
Do not just look at a green kubectl get pods — whether the sidecar has this configuration is the only evidence:
POD=$(kubectl -n <app-namespace> get pod -l app=<service> -o jsonpath='{.items[0].metadata.name}')
kubectl -n <app-namespace> exec "$POD" -c istio-proxy -- \
curl -s localhost:15000/config_dump | grep -B2 -A10 -i opentelemetry | head -40
Only when you see the opentelemetry exporter section is it actually connected. If you do not see it, the meshConfig did not take effect or the Pods were not restarted.
You do not need to expose Jaeger to the public internet to look at Traces; port forwarding plus an SSH tunnel is enough:
kubectl -n istio-system port-forward svc/tracing 16686:80
ssh -L 16686:127.0.0.1:16686 <user>@<host>
7. Post-install self-check
kubectl -n istio-system get pods # istiod and ingressgateway both Running
kubectl get gateway,vs -A # the entry point and routes are present
kubectl get svc -n istio-system istio-ingressgateway # note the NodePort for 80
curl -H 'Host: gateway.titanides.com' http://<node-ip>:<NodePort>/
That last line is the fastest check: anything other than a 404 means Host matching and routing both work; a 404 means go look at the spelling of the hosts
in vs, and a 503 means go look at the subset.