On-Premise Deployment
Run KugelAudio text-to-speech on premise on your own Kubernetes cluster: install, configure, start, stop, upgrade and roll back an Enterprise deployment.
On this page
This guide takes you from a supplied KugelAudio release to a running API, then
through stopping, upgrading, and recovering it. Speech synthesis runs on your
infrastructure. Your deployment's API still calls the hosted KugelAudio API
(api.kugelaudio.com) over outbound HTTPS for licensing, API-key
authentication, usage reporting, and voice lookups, so an outage of that path
affects request authentication, not only licensing.
The main path below is a single-machine k3s installation, with one TTS model, an optional normalizer, and standalone Valkey. A Kubernetes administrator must prepare the host and GPU runtime first. For an existing multi-node installation, use its supplied hardware profile and replica counts.
| Task | Go to |
|---|---|
| Install a new deployment | Prepare → Configure → Install |
| Connect your application | Set up API access |
| Start, stop, or restart | Control the application |
| Update to a new version | Upgrade |
| Restore the previous version | Roll back |
| Investigate a failure | Troubleshooting |
1. Prepare
Get your deployment package
Your KugelAudio contact supplies the approved chart package, checksum, deployment license, registry credentials, and reviewed hardware configuration. Contact sales if you have not arranged your Enterprise deployment yet. If you already have an agreement, ask your KugelAudio contact for these files and credentials.
Keep these in a working directory on the administration machine, one directory
per version (for example kugelaudio-<approved-version>/), so a new release's
SHA256SUMS never overwrites the one you need for rollback:
kugelaudio-onprem-<approved-version>.tgz
SHA256SUMS
values.customer.yaml
hardware.yamlThe two YAML files contain deployment settings; credentials are stored separately. Retain each accepted version's directory for recovery.
Prepare Kubernetes and the GPU
You need Linux, a supported Kubernetes/k3s version, Helm 3, kubectl, a working NVIDIA driver and container runtime, and the reviewed GPU device-plugin configuration. The application chart does not install these prerequisites.
For a new k3s host, follow the K3s installation instructions using the version agreed for your deployment. Configure the NVIDIA runtime and GPU add-on using the hardware instructions supplied with your release before continuing.
kubectl config current-context
kubectl get nodes
kubectl get runtimeclass
nvidia-smiReady to continue: the intended node is Ready, the NVIDIA runtime works, and Kubernetes advertises the GPU resources required by your hardware profile. The bundled single-GPU profile uses shared GPU slots; slots do not provide separate GPU memory.
On a k3s host, kubectl and Helm need an authorized kubeconfig. An administrator
can run them from a root shell with
KUBECONFIG=/etc/rancher/k3s/k3s.yaml; do not make that file world-readable.
Allow DNS and outbound HTTPS for the configured KugelAudio control plane, container registries, and image downloads listed in your handover. Fully air-gapped operation is not supported. Use your customer CA bundle if an HTTPS proxy is present.
Inspect the package
Run the remaining shell examples in Bash from your working directory. Replace the package placeholder with the actual filename:
export KUGELAUDIO_CHART="./kugelaudio-onprem-<approved-version>.tgz"
export KUGELAUDIO_RELEASE="kugelaudio"
export KUGELAUDIO_NAMESPACE="kugelaudio"
sha256sum --check SHA256SUMS # on macOS: shasum -a 256 -c SHA256SUMS
helm show chart "${KUGELAUDIO_CHART}"The package also contains INSTALL.md, README.md, and, from chart 1.0.2,
OPERATIONS.md. Read the matching installation prerequisites with:
tar -xOf "${KUGELAUDIO_CHART}" kugelaudio-onprem/INSTALL.md2. Configure
Create the deployment Secret
Create the namespace and save the chart's Secret template to a file:
kubectl create namespace "${KUGELAUDIO_NAMESPACE}"
tar -xOf "${KUGELAUDIO_CHART}" kugelaudio-onprem/examples/secrets.yaml > secrets.yamlReplace the placeholders in secrets.yaml with these values, and set
metadata.namespace to ${KUGELAUDIO_NAMESPACE} if you chose a namespace
other than kugelaudio:
| Secret field | Value |
|---|---|
.dockerconfigjson | Registry read credentials supplied by KugelAudio |
KUGEL_LICENSE_KEY | Deployment license supplied by KugelAudio |
TTS_MASTER_API_KEY | Customer-generated local operator key, for example from openssl rand -hex 32 |
Keep the Secret type kubernetes.io/dockerconfigjson and the default name
kugelaudio-onprem-secrets. The optional browser workspace needs the
additional credentials in the template. Apply the file, confirm the Secret
exists without displaying its values, then delete the plaintext file:
kubectl apply -f secrets.yaml
kubectl get secret kugelaudio-onprem-secrets -n "${KUGELAUDIO_NAMESPACE}"
rm secrets.yamlIf you manage secrets with SOPS, Sealed Secrets, or External Secrets, create
the same Secret through that workflow instead of kubectl apply.
Project API keys are created later for applications; they do not go into this Secret.
Set the model, capacity, and endpoint
Review the supplied values.customer.yaml and hardware.yaml together:
| Setting | What to decide |
|---|---|
secrets.existingSecret | Name of the Secret created above |
tts.models | Enabled model and replica count; one replica for the single-GPU profile |
tts.env.KUGELAUDIO_MAX_ACTIVE_TURNS | Agreed runtime capacity within the license and hardware limits. It caps the turns the GPU generates at once and also the sessions a replica keeps open, including idle conversations between turns |
normalizer.enabled | Whether to run the bundled neural normalizer |
valkey.ha.enabled | false for a single machine |
ingress.service.type | Private cluster service or an approved externally reachable service |
web.enabled | Optional browser workspace; not needed for API access |
File order matters: this guide applies customer values first and
hardware.yaml last. The hardware file wins on overlapping fields, including
concurrency and service exposure. Review the effective settings instead of
assuming a change in the first file overrides the profile.
Keep the approved GPU placement and memory settings. Changing a replica count does not add physical GPU capacity, and increasing the runtime limit does not increase a project's API-key limit or license entitlement.
The API Service is named ingress and serves port 8000. The single-GPU
preset defaults to ClusterIP; use an existing customer gateway or explicitly
configure supported exposure. LoadBalancer requires a working cluster
load-balancer implementation.
Keep a stable API URL for applications. A customer-managed proxy or gateway can preserve this URL when you switch deployments during an upgrade. Terminate TLS there when required by your network policy.
3. Install and verify
Installing the chart starts the application automatically:
helm install "${KUGELAUDIO_RELEASE}" "${KUGELAUDIO_CHART}" \
--namespace "${KUGELAUDIO_NAMESPACE}" \
-f ./values.customer.yaml \
-f ./hardware.yaml \
--atomic --wait --timeout 60mInitial image downloads, model loading, and warm-up can take several minutes. The timeout is a failure deadline, not an expected startup duration.
helm status "${KUGELAUDIO_RELEASE}" -n "${KUGELAUDIO_NAMESPACE}"
kubectl get pods,svc -n "${KUGELAUDIO_NAMESPACE}"
helm test "${KUGELAUDIO_RELEASE}" -n "${KUGELAUDIO_NAMESPACE}" --timeout 20mSuccess: the release is deployed, application containers are Ready, and the Helm test passes. This test checks health, readiness, and model catalogue; verify actual speech in the next step before accepting traffic.
Set the API URL assigned to your deployment:
export KUGELAUDIO_API_URL="https://tts.example.com"
curl --fail --connect-timeout 5 --max-time 10 "${KUGELAUDIO_API_URL}/health"Expect HTTP 200. If external routing is not ready, test locally with
kubectl port-forward -n "${KUGELAUDIO_NAMESPACE}" svc/ingress 8000:8000
and use http://127.0.0.1:8000 from another terminal. Port forwarding is a
temporary diagnostic, not the production endpoint.
4. Set up API access
Create a dedicated project API key in the KugelAudio Dashboard, following Getting your API key. This key authorizes generation; the deployment license and local operator key cannot replace it.
In Bash, load the key without recording its value in shell history:
read -r -s -p "Project API key: " KUGELAUDIO_API_KEY
printf '\n'
export KUGELAUDIO_API_KEYList voices available to this key:
curl --fail --connect-timeout 5 --max-time 30 \
"${KUGELAUDIO_API_URL}/v1/voices?limit=10" \
-H "Authorization: Bearer ${KUGELAUDIO_API_KEY}"Copy a voice's numeric id into the example below. See
voice selection and pagination if
you need a different voice.
curl --fail --connect-timeout 5 --max-time 120 \
"${KUGELAUDIO_API_URL}/v1/tts/generate" \
-H "Authorization: Bearer ${KUGELAUDIO_API_KEY}" \
-H "Content-Type: application/json" \
--data '{
"text": "Hello from my local KugelAudio deployment.",
"model_id": "kugel-3",
"voice_id": "<voice-id>",
"language": "en"
}' --output output.pcm &&
ffmpeg -f s16le -ar 24000 -ac 1 -i output.pcm output.wavOpen output.wav and confirm that you hear the supplied text. Conversion
requires FFmpeg; the Python SDK can save WAV directly.
See generation formats for raw audio.
Configure your application's SDK with this API key and the local base URL:
Python uses api_url; TypeScript and Java use apiUrl (see
SDK configuration). Store the key in server-side secret
configuration and keep caller concurrency within the agreed limits.
5. Start, stop, and restart
Install starts the services. There is no separate application daemon to launch.
The commands below apply to the single-machine profile with fixed replicas,
one kugel-3 TTS Deployment, and standalone Valkey in a namespace dedicated to
this release. If autoscaling or GitOps controls replicas, use that controller's
maintenance procedure; otherwise it may immediately undo a manual stop.
Stop
- Pause new requests at the caller or gateway.
- Let active generations and streaming connections finish, or close them deliberately within your agreed maintenance deadline.
- Scale the application Deployments down:
kubectl scale deployment --all --replicas=0 -n "${KUGELAUDIO_NAMESPACE}"
kubectl get deployments,pods -n "${KUGELAUDIO_NAMESPACE}"Stopped: all application Deployments have zero replicas and their running pods have terminated. Completed Helm-test pods can remain. GPU memory held by TTS and normalizer is released; standalone Valkey's transient state is lost. The release and its Secret remain installed.
Scaling is a temporary override. A later Helm update or start restores the replicas from the accepted values.
Start after a stop
Reapply the currently accepted package and values, restoring the recorded replicas. This creates a new Helm revision without selecting a newer release:
helm upgrade "${KUGELAUDIO_RELEASE}" "${KUGELAUDIO_CHART}" \
--namespace "${KUGELAUDIO_NAMESPACE}" \
-f ./values.customer.yaml -f ./hardware.yaml \
--wait --timeout 60mRepeat installation verification and the speech test, then resume callers. If startup fails, leave callers paused and inspect the failing pod before retrying.
Restart one component
For an unhealthy TTS component or a configuration change requiring a restart:
kubectl rollout restart deployment/tts-kugel-3 -n "${KUGELAUDIO_NAMESPACE}"
kubectl rollout status deployment/tts-kugel-3 \
-n "${KUGELAUDIO_NAMESPACE}" --timeout 20mUse deployment/ingress for the API or deployment/normalizer for the
enabled normalizer. Restarting TTS on a single GPU interrupts generation until
the replacement is ready. Verify health and speech before resuming callers.
6. Upgrade
This procedure updates the existing deployment on the same machine. Schedule a maintenance window: speech generation is unavailable while workloads are replaced and the model loads. The API URL stays the same.
Prepare the release
Obtain the approved new package and checksum into a new version directory. Read its changelog and review any required values changes; keep the old version's directory with its package and accepted values. From the new directory:
export KUGELAUDIO_NEW_CHART="$PWD/kugelaudio-onprem-<new-approved-version>.tgz"
sha256sum --check SHA256SUMS # on macOS: shasum -a 256 -c SHA256SUMS
helm show chart "${KUGELAUDIO_NEW_CHART}"
tar -xOf "${KUGELAUDIO_NEW_CHART}" kugelaudio-onprem/CHANGELOG.md
helm history "${KUGELAUDIO_RELEASE}" -n "${KUGELAUDIO_NAMESPACE}"Record the last accepted Helm revision as your rollback target. Confirm the current deployment can generate speech, the required images are reachable, and disk/GPU capacity is sufficient.
Apply the update
- Pause new requests and drain existing streams.
- Upgrade using the reviewed values:
helm upgrade "${KUGELAUDIO_RELEASE}" "${KUGELAUDIO_NEW_CHART}" \
--namespace "${KUGELAUDIO_NAMESPACE}" \
-f ./values.customer.yaml -f ./hardware.yaml \
--atomic --wait --timeout 60mVerify and resume
Keep callers paused and check the updated deployment:
helm status "${KUGELAUDIO_RELEASE}" -n "${KUGELAUDIO_NAMESPACE}"
kubectl get pods -n "${KUGELAUDIO_NAMESPACE}"
helm test "${KUGELAUDIO_RELEASE}" -n "${KUGELAUDIO_NAMESPACE}" --timeout 20m
curl --fail --connect-timeout 5 --max-time 10 "${KUGELAUDIO_API_URL}/health"Repeat the voice selection and speech request through your deployment's API URL and listen to the result. Resume callers only when application containers are Ready, the Helm test and health request pass, and speech works with the voices and languages your application uses.
After accepting the update, use its package for future starts:
export KUGELAUDIO_CHART="${KUGELAUDIO_NEW_CHART}"
helm history "${KUGELAUDIO_RELEASE}" -n "${KUGELAUDIO_NAMESPACE}"Record that package path, its values files, and the new Helm revision in your deployment handover. Keep the previous package and values for rollback.
Do not override individual image/weight versions or use --reuse-values.
Helm 3's --atomic rolls back a failed upgrade, but passing pod readiness does
not prove speech works. If the speech test fails, use rollback
while callers remain paused.
7. Roll back
Keep callers paused. Use the revision recorded before the update:
helm history "${KUGELAUDIO_RELEASE}" -n "${KUGELAUDIO_NAMESPACE}"
helm rollback "${KUGELAUDIO_RELEASE}" <accepted-revision> \
-n "${KUGELAUDIO_NAMESPACE}" --wait --timeout 60mRepeat the verification checks and speech test, then resume callers. Reset your operations record to the restored package and values so the next start does not inadvertently reapply the rejected release.
Rollback restores Helm-managed resources. It does not restore an externally changed Secret, customer CA, GPU driver, gateway configuration, or lost transient state. Old images and valid licensing must still be available.
Troubleshooting
Start with the failing workload:
kubectl get pods -n "${KUGELAUDIO_NAMESPACE}"
kubectl get events -n "${KUGELAUDIO_NAMESPACE}" --sort-by='.lastTimestamp'
kubectl logs <pod-name> -n "${KUGELAUDIO_NAMESPACE}" \
--all-containers --since=15m| Problem | First check |
|---|---|
| Pod Pending | GPU allocation, CPU/RAM, node placement, and spare capacity |
| Image pull or license failure | Registry credential, license, DNS, outbound HTTPS, and proxy CA |
| Health works locally but not from the application | Service exposure, customer routing, firewall, and TLS |
| Authentication or voice failure | API key troubleshooting, voice selection, and outbound HTTPS to api.kugelaudio.com |
| API-key authentication fails for every key | The hosted API is unreachable: check DNS, firewall, proxy CA, and outbound HTTPS to api.kugelaudio.com from the ingress pod |
| GPU process crashes | nvidia-smi, driver state, and the affected pod's logs |
| Upgrade passes but speech fails | Keep traffic paused; inspect logs and roll back if necessary |
For routine maintenance, monitor pod restarts, GPU and disk health, the API health endpoint, and an authenticated speech probe. Keep the chart, values, hardware profile, and matching operations manual available; back up credentials through your secret manager. A single machine has no node redundancy.
For support, include the chart version, Helm revision, timestamp, failed check, and relevant reviewed logs. Exclude credential values, environment dumps, and signed URLs. Contact KugelAudio.