Skip to content
KugelAudioDocs

On-Premise Deployment

Run KugelAudio text-to-speech on premise on your own Kubernetes cluster: install, configure, start, stop, upgrade and roll back an Enterprise deployment.

On this page

This guide takes you from a supplied KugelAudio release to a running API, then through stopping, upgrading, and recovering it. Speech synthesis runs on your infrastructure. Your deployment's API still calls the hosted KugelAudio API (api.kugelaudio.com) over outbound HTTPS for licensing, API-key authentication, usage reporting, and voice lookups, so an outage of that path affects request authentication, not only licensing.

The main path below is a single-machine k3s installation, with one TTS model, an optional normalizer, and standalone Valkey. A Kubernetes administrator must prepare the host and GPU runtime first. For an existing multi-node installation, use its supplied hardware profile and replica counts.

TaskGo to
Install a new deploymentPrepare → Configure → Install
Connect your applicationSet up API access
Start, stop, or restartControl the application
Update to a new versionUpgrade
Restore the previous versionRoll back
Investigate a failureTroubleshooting

1. Prepare

Get your deployment package

Your KugelAudio contact supplies the approved chart package, checksum, deployment license, registry credentials, and reviewed hardware configuration. Contact sales if you have not arranged your Enterprise deployment yet. If you already have an agreement, ask your KugelAudio contact for these files and credentials.

Keep these in a working directory on the administration machine, one directory per version (for example kugelaudio-<approved-version>/), so a new release's SHA256SUMS never overwrites the one you need for rollback:

Text
kugelaudio-onprem-<approved-version>.tgz
SHA256SUMS
values.customer.yaml
hardware.yaml

The two YAML files contain deployment settings; credentials are stored separately. Retain each accepted version's directory for recovery.

Prepare Kubernetes and the GPU

You need Linux, a supported Kubernetes/k3s version, Helm 3, kubectl, a working NVIDIA driver and container runtime, and the reviewed GPU device-plugin configuration. The application chart does not install these prerequisites.

For a new k3s host, follow the K3s installation instructions using the version agreed for your deployment. Configure the NVIDIA runtime and GPU add-on using the hardware instructions supplied with your release before continuing.

Bash
kubectl config current-context
kubectl get nodes
kubectl get runtimeclass
nvidia-smi

Ready to continue: the intended node is Ready, the NVIDIA runtime works, and Kubernetes advertises the GPU resources required by your hardware profile. The bundled single-GPU profile uses shared GPU slots; slots do not provide separate GPU memory.

On a k3s host, kubectl and Helm need an authorized kubeconfig. An administrator can run them from a root shell with KUBECONFIG=/etc/rancher/k3s/k3s.yaml; do not make that file world-readable.

Allow DNS and outbound HTTPS for the configured KugelAudio control plane, container registries, and image downloads listed in your handover. Fully air-gapped operation is not supported. Use your customer CA bundle if an HTTPS proxy is present.

Inspect the package

Run the remaining shell examples in Bash from your working directory. Replace the package placeholder with the actual filename:

Bash
export KUGELAUDIO_CHART="./kugelaudio-onprem-<approved-version>.tgz"
export KUGELAUDIO_RELEASE="kugelaudio"
export KUGELAUDIO_NAMESPACE="kugelaudio"

sha256sum --check SHA256SUMS   # on macOS: shasum -a 256 -c SHA256SUMS
helm show chart "${KUGELAUDIO_CHART}"

The package also contains INSTALL.md, README.md, and, from chart 1.0.2, OPERATIONS.md. Read the matching installation prerequisites with:

Bash
tar -xOf "${KUGELAUDIO_CHART}" kugelaudio-onprem/INSTALL.md

2. Configure

Create the deployment Secret

Create the namespace and save the chart's Secret template to a file:

Bash
kubectl create namespace "${KUGELAUDIO_NAMESPACE}"
tar -xOf "${KUGELAUDIO_CHART}" kugelaudio-onprem/examples/secrets.yaml > secrets.yaml

Replace the placeholders in secrets.yaml with these values, and set metadata.namespace to ${KUGELAUDIO_NAMESPACE} if you chose a namespace other than kugelaudio:

Secret fieldValue
.dockerconfigjsonRegistry read credentials supplied by KugelAudio
KUGEL_LICENSE_KEYDeployment license supplied by KugelAudio
TTS_MASTER_API_KEYCustomer-generated local operator key, for example from openssl rand -hex 32

Keep the Secret type kubernetes.io/dockerconfigjson and the default name kugelaudio-onprem-secrets. The optional browser workspace needs the additional credentials in the template. Apply the file, confirm the Secret exists without displaying its values, then delete the plaintext file:

Bash
kubectl apply -f secrets.yaml
kubectl get secret kugelaudio-onprem-secrets -n "${KUGELAUDIO_NAMESPACE}"
rm secrets.yaml

If you manage secrets with SOPS, Sealed Secrets, or External Secrets, create the same Secret through that workflow instead of kubectl apply.

Project API keys are created later for applications; they do not go into this Secret.

Set the model, capacity, and endpoint

Review the supplied values.customer.yaml and hardware.yaml together:

SettingWhat to decide
secrets.existingSecretName of the Secret created above
tts.modelsEnabled model and replica count; one replica for the single-GPU profile
tts.env.KUGELAUDIO_MAX_ACTIVE_TURNSAgreed runtime capacity within the license and hardware limits. It caps the turns the GPU generates at once and also the sessions a replica keeps open, including idle conversations between turns
normalizer.enabledWhether to run the bundled neural normalizer
valkey.ha.enabledfalse for a single machine
ingress.service.typePrivate cluster service or an approved externally reachable service
web.enabledOptional browser workspace; not needed for API access

File order matters: this guide applies customer values first and hardware.yaml last. The hardware file wins on overlapping fields, including concurrency and service exposure. Review the effective settings instead of assuming a change in the first file overrides the profile.

Keep the approved GPU placement and memory settings. Changing a replica count does not add physical GPU capacity, and increasing the runtime limit does not increase a project's API-key limit or license entitlement.

The API Service is named ingress and serves port 8000. The single-GPU preset defaults to ClusterIP; use an existing customer gateway or explicitly configure supported exposure. LoadBalancer requires a working cluster load-balancer implementation.

Keep a stable API URL for applications. A customer-managed proxy or gateway can preserve this URL when you switch deployments during an upgrade. Terminate TLS there when required by your network policy.

3. Install and verify

Installing the chart starts the application automatically:

Bash
helm install "${KUGELAUDIO_RELEASE}" "${KUGELAUDIO_CHART}" \
  --namespace "${KUGELAUDIO_NAMESPACE}" \
  -f ./values.customer.yaml \
  -f ./hardware.yaml \
  --atomic --wait --timeout 60m

Initial image downloads, model loading, and warm-up can take several minutes. The timeout is a failure deadline, not an expected startup duration.

Bash
helm status "${KUGELAUDIO_RELEASE}" -n "${KUGELAUDIO_NAMESPACE}"
kubectl get pods,svc -n "${KUGELAUDIO_NAMESPACE}"
helm test "${KUGELAUDIO_RELEASE}" -n "${KUGELAUDIO_NAMESPACE}" --timeout 20m

Success: the release is deployed, application containers are Ready, and the Helm test passes. This test checks health, readiness, and model catalogue; verify actual speech in the next step before accepting traffic.

Set the API URL assigned to your deployment:

Bash
export KUGELAUDIO_API_URL="https://tts.example.com"
curl --fail --connect-timeout 5 --max-time 10 "${KUGELAUDIO_API_URL}/health"

Expect HTTP 200. If external routing is not ready, test locally with kubectl port-forward -n "${KUGELAUDIO_NAMESPACE}" svc/ingress 8000:8000 and use http://127.0.0.1:8000 from another terminal. Port forwarding is a temporary diagnostic, not the production endpoint.

4. Set up API access

Create a dedicated project API key in the KugelAudio Dashboard, following Getting your API key. This key authorizes generation; the deployment license and local operator key cannot replace it.

In Bash, load the key without recording its value in shell history:

Bash
read -r -s -p "Project API key: " KUGELAUDIO_API_KEY
printf '\n'
export KUGELAUDIO_API_KEY

List voices available to this key:

Bash
curl --fail --connect-timeout 5 --max-time 30 \
  "${KUGELAUDIO_API_URL}/v1/voices?limit=10" \
  -H "Authorization: Bearer ${KUGELAUDIO_API_KEY}"

Copy a voice's numeric id into the example below. See voice selection and pagination if you need a different voice.

Bash
curl --fail --connect-timeout 5 --max-time 120 \
  "${KUGELAUDIO_API_URL}/v1/tts/generate" \
  -H "Authorization: Bearer ${KUGELAUDIO_API_KEY}" \
  -H "Content-Type: application/json" \
  --data '{
    "text": "Hello from my local KugelAudio deployment.",
    "model_id": "kugel-3",
    "voice_id": "<voice-id>",
    "language": "en"
  }' --output output.pcm &&
ffmpeg -f s16le -ar 24000 -ac 1 -i output.pcm output.wav

Open output.wav and confirm that you hear the supplied text. Conversion requires FFmpeg; the Python SDK can save WAV directly. See generation formats for raw audio.

Configure your application's SDK with this API key and the local base URL: Python uses api_url; TypeScript and Java use apiUrl (see SDK configuration). Store the key in server-side secret configuration and keep caller concurrency within the agreed limits.

5. Start, stop, and restart

Install starts the services. There is no separate application daemon to launch.

The commands below apply to the single-machine profile with fixed replicas, one kugel-3 TTS Deployment, and standalone Valkey in a namespace dedicated to this release. If autoscaling or GitOps controls replicas, use that controller's maintenance procedure; otherwise it may immediately undo a manual stop.

Stop

  1. Pause new requests at the caller or gateway.
  2. Let active generations and streaming connections finish, or close them deliberately within your agreed maintenance deadline.
  3. Scale the application Deployments down:
Bash
kubectl scale deployment --all --replicas=0 -n "${KUGELAUDIO_NAMESPACE}"
kubectl get deployments,pods -n "${KUGELAUDIO_NAMESPACE}"

Stopped: all application Deployments have zero replicas and their running pods have terminated. Completed Helm-test pods can remain. GPU memory held by TTS and normalizer is released; standalone Valkey's transient state is lost. The release and its Secret remain installed.

Scaling is a temporary override. A later Helm update or start restores the replicas from the accepted values.

Start after a stop

Reapply the currently accepted package and values, restoring the recorded replicas. This creates a new Helm revision without selecting a newer release:

Bash
helm upgrade "${KUGELAUDIO_RELEASE}" "${KUGELAUDIO_CHART}" \
  --namespace "${KUGELAUDIO_NAMESPACE}" \
  -f ./values.customer.yaml -f ./hardware.yaml \
  --wait --timeout 60m

Repeat installation verification and the speech test, then resume callers. If startup fails, leave callers paused and inspect the failing pod before retrying.

Restart one component

For an unhealthy TTS component or a configuration change requiring a restart:

Bash
kubectl rollout restart deployment/tts-kugel-3 -n "${KUGELAUDIO_NAMESPACE}"
kubectl rollout status deployment/tts-kugel-3 \
  -n "${KUGELAUDIO_NAMESPACE}" --timeout 20m

Use deployment/ingress for the API or deployment/normalizer for the enabled normalizer. Restarting TTS on a single GPU interrupts generation until the replacement is ready. Verify health and speech before resuming callers.

6. Upgrade

This procedure updates the existing deployment on the same machine. Schedule a maintenance window: speech generation is unavailable while workloads are replaced and the model loads. The API URL stays the same.

Prepare the release

Obtain the approved new package and checksum into a new version directory. Read its changelog and review any required values changes; keep the old version's directory with its package and accepted values. From the new directory:

Bash
export KUGELAUDIO_NEW_CHART="$PWD/kugelaudio-onprem-<new-approved-version>.tgz"
sha256sum --check SHA256SUMS   # on macOS: shasum -a 256 -c SHA256SUMS
helm show chart "${KUGELAUDIO_NEW_CHART}"
tar -xOf "${KUGELAUDIO_NEW_CHART}" kugelaudio-onprem/CHANGELOG.md
helm history "${KUGELAUDIO_RELEASE}" -n "${KUGELAUDIO_NAMESPACE}"

Record the last accepted Helm revision as your rollback target. Confirm the current deployment can generate speech, the required images are reachable, and disk/GPU capacity is sufficient.

Apply the update

  1. Pause new requests and drain existing streams.
  2. Upgrade using the reviewed values:
Bash
helm upgrade "${KUGELAUDIO_RELEASE}" "${KUGELAUDIO_NEW_CHART}" \
  --namespace "${KUGELAUDIO_NAMESPACE}" \
  -f ./values.customer.yaml -f ./hardware.yaml \
  --atomic --wait --timeout 60m

Verify and resume

Keep callers paused and check the updated deployment:

Bash
helm status "${KUGELAUDIO_RELEASE}" -n "${KUGELAUDIO_NAMESPACE}"
kubectl get pods -n "${KUGELAUDIO_NAMESPACE}"
helm test "${KUGELAUDIO_RELEASE}" -n "${KUGELAUDIO_NAMESPACE}" --timeout 20m
curl --fail --connect-timeout 5 --max-time 10 "${KUGELAUDIO_API_URL}/health"

Repeat the voice selection and speech request through your deployment's API URL and listen to the result. Resume callers only when application containers are Ready, the Helm test and health request pass, and speech works with the voices and languages your application uses.

After accepting the update, use its package for future starts:

Bash
export KUGELAUDIO_CHART="${KUGELAUDIO_NEW_CHART}"
helm history "${KUGELAUDIO_RELEASE}" -n "${KUGELAUDIO_NAMESPACE}"

Record that package path, its values files, and the new Helm revision in your deployment handover. Keep the previous package and values for rollback.

Do not override individual image/weight versions or use --reuse-values. Helm 3's --atomic rolls back a failed upgrade, but passing pod readiness does not prove speech works. If the speech test fails, use rollback while callers remain paused.

7. Roll back

Keep callers paused. Use the revision recorded before the update:

Bash
helm history "${KUGELAUDIO_RELEASE}" -n "${KUGELAUDIO_NAMESPACE}"
helm rollback "${KUGELAUDIO_RELEASE}" <accepted-revision> \
  -n "${KUGELAUDIO_NAMESPACE}" --wait --timeout 60m

Repeat the verification checks and speech test, then resume callers. Reset your operations record to the restored package and values so the next start does not inadvertently reapply the rejected release.

Rollback restores Helm-managed resources. It does not restore an externally changed Secret, customer CA, GPU driver, gateway configuration, or lost transient state. Old images and valid licensing must still be available.

Troubleshooting

Start with the failing workload:

Bash
kubectl get pods -n "${KUGELAUDIO_NAMESPACE}"
kubectl get events -n "${KUGELAUDIO_NAMESPACE}" --sort-by='.lastTimestamp'
kubectl logs <pod-name> -n "${KUGELAUDIO_NAMESPACE}" \
  --all-containers --since=15m
ProblemFirst check
Pod PendingGPU allocation, CPU/RAM, node placement, and spare capacity
Image pull or license failureRegistry credential, license, DNS, outbound HTTPS, and proxy CA
Health works locally but not from the applicationService exposure, customer routing, firewall, and TLS
Authentication or voice failureAPI key troubleshooting, voice selection, and outbound HTTPS to api.kugelaudio.com
API-key authentication fails for every keyThe hosted API is unreachable: check DNS, firewall, proxy CA, and outbound HTTPS to api.kugelaudio.com from the ingress pod
GPU process crashesnvidia-smi, driver state, and the affected pod's logs
Upgrade passes but speech failsKeep traffic paused; inspect logs and roll back if necessary

For routine maintenance, monitor pod restarts, GPU and disk health, the API health endpoint, and an authenticated speech probe. Keep the chart, values, hardware profile, and matching operations manual available; back up credentials through your secret manager. A single machine has no node redundancy.

For support, include the chart version, Helm revision, timestamp, failed check, and relevant reviewed logs. Exclude credential values, environment dumps, and signed URLs. Contact KugelAudio.

Next steps