Lab 16: Serve Models from a KitOps ModelKit on HAMi

AdvancedDuration: about 60 minutesEnvironment: Kubernetes cluster with NVIDIA GPUsVerified: 2026-07-23By: @rudrakshkarpe, @shivaylamba

This lab demonstrates how to package a model as a KitOps ModelKit, a versioned OCI artifact, and download it from an OCI registry (Jozu Hub in the examples) into the Pod using a KitOps initContainer, then serve it from a local directory with SGLang (primary) or vLLM (optional co-resident example) on HAMi-virtualized GPU shares.

Like Lab 6 (vLLM), the inference engines run on HAMi resources. Here the model supply chain is registry-native: the model is packaged as a ModelKit, versioned and stored on Jozu Hub, pulled into the Pod as an OCI artifact, and served from a local path.

Learning Objectives

• Inspect a public KitOps ModelKit on an OCI registry
• Build a small kitunpacker init image and a custom SGLang serve image
• Deploy a Pod that pulls/unpacks a ModelKit via initContainer (the main container loads the model from the KitOps-delivered volume)
• Schedule the workload with HAMi nvidia.com/gpu / gpumem / gpucores
• Prove inference works against the OpenAI-compatible SGLang API
• Optionally co-locate a vLLM Pod serving the same ModelKit on the same physical GPU

Lab Overview

HAMi + KitOps ModelKit Lab Flowchart

Deployment Architecture

KitOps ModelKit delivery on HAMi

Prerequisites

• A Kubernetes cluster with NVIDIA GPUs, HAMi installed and healthy, kubectl, and helm (see Lab 6 for a complete setup)
• Docker (or an equivalent builder) to build and load images into the cluster
• kit CLI on your workstation (optional but recommended for kit inspect)
• Ability to pull from the public registry jozu.ml (no login required for the sample ModelKit)

This lab assumes HAMi is already installed. If not, complete Steps 1–3 of Lab 6 first.

Example Cluster State

Verification used a kind + H100 cluster with HAMi advertising 10 vGPUs and both hami-scheduler / hami-device-plugin Running.

Public ModelKit used throughout:

jozu.ml/jonathangamer202002/qwen3-4b-instruct@sha256:df4629f6a10bba7bec45e12bd15f910ed1024699bfbb44b63240899f71bb1c19

Step 1: Confirm HAMi Is Ready

kubectl get pods -n kube-system -l app.kubernetes.io/instance=hami -o wide
kubectl get nodes -o 'custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'

Expected: device plugin and scheduler Running; GPU nodes show allocatable nvidia.com/gpu (for example 10).

Check the workloads currently using GPU shares and scale down any workloads that are not needed for this lab. The ModelKit and 4B model need a 30 GiB HAMi memory share in the example below.

kubectl get pods --all-namespaces \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,GPU-MEM:.spec.containers[*].resources.limits.nvidia\.com/gpumem'

Step 2: Inspect the ModelKit

On a machine with the kit CLI:

kit inspect --remote jozu.ml/jonathangamer202002/qwen3-4b-instruct@sha256:df4629f6a10bba7bec45e12bd15f910ed1024699bfbb44b63240899f71bb1c19

Example (truncated) output from the verification environment:

{
  "digest": "sha256:df4629f6a10bba7bec45e12bd15f910ed1024699bfbb44b63240899f71bb1c19",
  "kitfile": {
    "package": { "name": "Qwen3-4B-Instruct-2507", "version": "1.0" },
    "model": {
      "name": "qwen3-4b-instruct",
      "path": "qwen3-4b-instruct/model",
      "license": "Apache 2.0"
    }
  },
  "manifest": {
    "artifactType": "application/vnd.kitops.modelkit.manifest.v1+json"
  }
}

The ModelKit carries safetensors weights plus tokenizer/config as OCI layers. The initContainer will unpack and flatten them into a flat model directory (config.json + *.safetensors) that SGLang/vLLM can load locally.

Step 3: Build the Pipeline Images

Create a working directory and the files below.

3.1 kitunpacker init image

kitunpacker/Dockerfile:

FROM alpine:3.20

ARG KITOPS_VERSION=v1.11.0
ARG TARGETARCH

RUN apk add --no-cache bash coreutils findutils ca-certificates curl tar \
    && case "${TARGETARCH}" in amd64) KITOPS_ARCH=x86_64 ;; arm64) KITOPS_ARCH=arm64 ;; *) echo "unsupported architecture: ${TARGETARCH}" >&2; exit 1 ;; esac \
    && KITOPS_ASSET="kitops-linux-${KITOPS_ARCH}.tar.gz" \
    && curl -fsSL "https://github.com/kitops-ml/kitops/releases/download/${KITOPS_VERSION}/${KITOPS_ASSET}" -o /tmp/kit.tgz \
    && curl -fsSL "https://github.com/kitops-ml/kitops/releases/download/${KITOPS_VERSION}/kitops_${KITOPS_VERSION}_checksums.txt" -o /tmp/kit-checksums.txt \
    && grep "  ${KITOPS_ASSET}$" /tmp/kit-checksums.txt | sed "s#${KITOPS_ASSET}#/tmp/kit.tgz#" | sha256sum -c - \
    && tar -xzf /tmp/kit.tgz -C /usr/local/bin kit \
    && rm -f /tmp/kit.tgz /tmp/kit-checksums.txt \
    && kit version

ENV MODELKIT_REF="jozu.ml/jonathangamer202002/qwen3-4b-instruct@sha256:df4629f6a10bba7bec45e12bd15f910ed1024699bfbb44b63240899f71bb1c19" \
    UNPACK_PATH="/models" \
    MODEL_SUBDIR="qwen3"

COPY unpack.sh /usr/local/bin/unpack.sh
RUN chmod +x /usr/local/bin/unpack.sh

ENTRYPOINT ["/usr/local/bin/unpack.sh"]

kitunpacker/unpack.sh:

#!/usr/bin/env sh
# kitunpacker: pull a ModelKit from an OCI registry (Jozu Hub by default) and
# unpack the model into a flat directory (config.json + *.safetensors) that
# vLLM / SGLang can load directly from local disk.
#
# Env (all overridable from the Pod spec):
#   MODELKIT_REF   full ModelKit reference, e.g. jozu.ml/<org>/<repo>:<tag>
#   UNPACK_PATH    mounted model root, restricted to /models  (default /models)
#   MODEL_SUBDIR   final model dir under UNPACK_PATH          (default qwen3)
#   REGISTRY_URL/USERNAME/PASSWORD  optional creds for PRIVATE registries
set -eu

MODELKIT_REF="${MODELKIT_REF:?MODELKIT_REF is required}"
UNPACK_PATH="${UNPACK_PATH:-/models}"
MODEL_SUBDIR="${MODEL_SUBDIR:-qwen3}"

[ "${UNPACK_PATH}" = "/models" ] || {
  echo "[kitunpacker] UNPACK_PATH must be /models" >&2
  exit 1
}
case "${MODEL_SUBDIR}" in
  "" | "." | ".." | */* | *[!A-Za-z0-9._-]*)
    echo "[kitunpacker] MODEL_SUBDIR must be one safe path component" >&2
    exit 1
    ;;
esac

REF_KEY="$(printf '%s' "${MODELKIT_REF}" | sha256sum | cut -d ' ' -f 1)"
RELEASES="${UNPACK_PATH}/.releases-${MODEL_SUBDIR}"
PUBLISHED="${RELEASES}/${REF_KEY}"
DEST="${UNPACK_PATH}/${MODEL_SUBDIR}"
RAW="${UNPACK_PATH}/.raw-${MODEL_SUBDIR}"
STAGE="${RELEASES}/.stage-${REF_KEY}-$$"
LINK_TMP="${UNPACK_PATH}/.link-${MODEL_SUBDIR}-$$"
LOCK="${UNPACK_PATH}/.lock-${MODEL_SUBDIR}"
MARKER=".modelkit-ref"
LOCK_OWNER="${HOSTNAME:-pod}-$$"
STALE_LOCK_SECONDS=600

# keep the kit pull cache on the (large) mounted volume, not the tiny rootfs
export KITOPS_HOME="${UNPACK_PATH}/.kitcache"

valid_model() {
  [ -f "$1/config.json" ] && ls "$1"/*.safetensors >/dev/null 2>&1
}

ready() {
  valid_model "${DEST}" &&
    [ -f "${DEST}/${MARKER}" ] &&
    [ "$(cat "${DEST}/${MARKER}")" = "${MODELKIT_REF}" ]
}

echo "[kitunpacker] ref=${MODELKIT_REF} -> ${DEST}"

if ready; then
  echo "[kitunpacker] model already present, skipping unpack"
  exit 0
fi

# This lock only coordinates Pods when they mount the same shared PVC. With
# the emptyDir used in this lab, every Pod has an isolated volume and lock.
# mkdir is atomic. A heartbeat allows recovery when a Pod is killed after
# acquiring the lock, and each waiter retries acquisition after contention.
acquire_lock() {
  mkdir "${LOCK}" 2>/dev/null || return 1
  printf '%s\n' "${LOCK_OWNER}" >"${LOCK}/owner"
  touch "${LOCK}/heartbeat"
}

lock_is_stale() {
  target="${LOCK}/heartbeat"
  [ -e "${target}" ] || target="${LOCK}"
  now="$(date +%s)"
  modified="$(stat -c %Y "${target}" 2>/dev/null || echo "${now}")"
  [ $((now - modified)) -gt "${STALE_LOCK_SECONDS}" ]
}

i=0
until acquire_lock; do
  ready && { echo "[kitunpacker] model became ready"; exit 0; }
  if lock_is_stale; then
    stale="${UNPACK_PATH}/.stale-lock-${MODEL_SUBDIR}-${LOCK_OWNER}"
    if mv "${LOCK}" "${stale}" 2>/dev/null; then
      echo "[kitunpacker] recovered stale lock"
      rm -rf "${stale}"
      continue
    fi
  fi
  [ "${i}" -lt 360 ] || {
    echo "[kitunpacker] timed out waiting for peer unpack" >&2
    exit 1
  }
  i=$((i + 1))
  sleep 5
done

heartbeat() {
  while touch "${LOCK}/heartbeat" 2>/dev/null; do sleep 15; done
}
heartbeat &
HEARTBEAT_PID=$!

cleanup() {
  status=$?
  trap - EXIT
  kill "${HEARTBEAT_PID}" 2>/dev/null || true
  wait "${HEARTBEAT_PID}" 2>/dev/null || true
  rm -rf "${RAW}" "${STAGE}" "${LINK_TMP}" "${KITOPS_HOME}"
  if [ "$(cat "${LOCK}/owner" 2>/dev/null || true)" = "${LOCK_OWNER}" ]; then
    rm -rf "${LOCK}"
  fi
  exit "${status}"
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

# optional login for private registries (public Jozu Hub needs none)
if [ -n "${REGISTRY_URL:-}" ] && [ -n "${USERNAME:-}" ] && [ -n "${PASSWORD:-}" ]; then
  echo "[kitunpacker] logging in to ${REGISTRY_URL} as ${USERNAME}"
  echo "${PASSWORD}" | kit login "${REGISTRY_URL}" -u "${USERNAME}" --password-stdin
fi

rm -rf "${RAW}" "${STAGE}"
mkdir -p "${RAW}" "${STAGE}" "${RELEASES}"
echo "[kitunpacker] pulling + unpacking model layers from registry..."
kit unpack --filter model "${MODELKIT_REF}" -d "${RAW}"

# Flatten: ModelKits may store the .safetensors shards in a model/ subdir while
# config.json / *.index.json / tokenizer sit one level up. vLLM/transformers
# need them all in one directory, so collect everything into a staging directory.
CONFIGS="$(find "${RAW}" -type f -name config.json -print)"
CONFIG_COUNT="$(printf '%s\n' "${CONFIGS}" | sed '/^$/d' | wc -l | tr -d ' ')"
[ "${CONFIG_COUNT}" -eq 1 ] || {
  echo "[kitunpacker] expected exactly one config.json, found ${CONFIG_COUNT}" >&2
  exit 1
}
SRC_CFG="${CONFIGS}"
SRC="$(dirname "${SRC_CFG}")"

# all weight shards, wherever they live under the unpacked tree
find "${SRC}" -name '*.safetensors' -exec mv -f {} "${STAGE}/" \;
# all top-level metadata files (config, index, tokenizer, vocab, generation cfg)
find "${SRC}" -maxdepth 1 -type f -exec mv -f {} "${STAGE}/" \;

valid_model "${STAGE}" || { echo "[kitunpacker] validation failed: missing config or shards" >&2; exit 1; }
printf '%s\n' "${MODELKIT_REF}" >"${STAGE}/${MARKER}"

# Publish an immutable, validated release directory, then atomically switch the
# destination symlink. Existing readers can keep using the previous directory.
if [ ! -d "${PUBLISHED}" ]; then
  mv "${STAGE}" "${PUBLISHED}"
fi
[ ! -e "${DEST}" ] || [ -L "${DEST}" ] || {
  echo "[kitunpacker] refusing to replace non-symlink destination ${DEST}" >&2
  exit 1
}
ln -s "${PUBLISHED}" "${LINK_TMP}"
mv -Tf "${LINK_TMP}" "${DEST}"

echo "[kitunpacker] final model directory:"
ls -la "${DEST}"
ready || { echo "[kitunpacker] publication validation failed" >&2; exit 1; }
echo "[kitunpacker] done."

Build:

docker build -t hami-kitunpacker:latest ./kitunpacker

3.2 Custom SGLang image (serves a local model path)

sglang/Dockerfile:

FROM lmsysorg/sglang:v0.5.7

ENV MODEL_DIR="/models/qwen3" \
    SERVED_NAME="qwen3-4b-instruct" \
    CONTEXT_LEN="8192" \
    MEM_FRACTION="0.8" \
    PORT="30000" \
    ATTENTION_BACKEND="triton"

COPY serve.sh /usr/local/bin/serve.sh
RUN chmod +x /usr/local/bin/serve.sh

ENTRYPOINT ["/usr/local/bin/serve.sh"]

sglang/serve.sh:

#!/usr/bin/env bash
# Custom SGLang entrypoint: serve a model unpacked from a KitOps ModelKit that
# the kitunpacker initContainer placed on a shared volume. Serves from a LOCAL
# directory (--model-path) delivered straight from the Jozu Hub ModelKit.
set -euo pipefail

MODEL_DIR="${MODEL_DIR:-/models/qwen3}"
SERVED_NAME="${SERVED_NAME:-qwen3-4b-instruct}"
CONTEXT_LEN="${CONTEXT_LEN:-8192}"
MEM_FRACTION="${MEM_FRACTION:-0.8}"
PORT="${PORT:-30000}"
ATTENTION_BACKEND="${ATTENTION_BACKEND:-triton}"

echo "[sglang-jozu] serving KitOps model from ${MODEL_DIR} (source: Jozu Hub ModelKit)"
if [ ! -f "${MODEL_DIR}/config.json" ]; then
  echo "[sglang-jozu] ERROR: ${MODEL_DIR}/config.json not found -- did the kitunpacker init run?" >&2
  exit 1
fi

exec python3 -m sglang.launch_server \
  --model-path "${MODEL_DIR}" \
  --served-model-name "${SERVED_NAME}" \
  --host 0.0.0.0 \
  --port "${PORT}" \
  --mem-fraction-static "${MEM_FRACTION}" \
  --context-length "${CONTEXT_LEN}" \
  --attention-backend "${ATTENTION_BACKEND}"

Build:

docker build -t hami-sglang-jozu:latest ./sglang

3.3 Load images into the cluster

For kind:

kind load docker-image hami-kitunpacker:latest --name <your-cluster>
kind load docker-image hami-sglang-jozu:latest --name <your-cluster>

For other clusters, push the images to a registry your nodes can pull and update the Deployment image fields accordingly.

3.4 Optional: custom vLLM image (for Step 8)

vllm/Dockerfile:

FROM vllm/vllm-openai:v0.23.0

ENV MODEL_DIR="/models/qwen3" \
    SERVED_NAME="qwen3-4b-instruct" \
    MAX_MODEL_LEN="8192" \
    GPU_MEM_UTIL="0.85" \
    PORT="8000"

COPY serve.sh /usr/local/bin/serve.sh
RUN chmod +x /usr/local/bin/serve.sh

ENTRYPOINT ["/usr/local/bin/serve.sh"]

vllm/serve.sh:

#!/usr/bin/env bash
# Custom vLLM entrypoint: serve a model unpacked from a KitOps ModelKit that the
# kitunpacker initContainer placed on a shared volume. It serves from a LOCAL
# directory (--model-path) populated from the Jozu Hub ModelKit.
set -euo pipefail

MODEL_DIR="${MODEL_DIR:-/models/qwen3}"
SERVED_NAME="${SERVED_NAME:-qwen3-4b-instruct}"
MAX_MODEL_LEN="${MAX_MODEL_LEN:-8192}"
GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.85}"
PORT="${PORT:-8000}"

echo "[vllm-jozu] serving KitOps model from ${MODEL_DIR} (source: Jozu Hub ModelKit)"
if [ ! -f "${MODEL_DIR}/config.json" ]; then
  echo "[vllm-jozu] ERROR: ${MODEL_DIR}/config.json not found -- did the kitunpacker init run?" >&2
  exit 1
fi

exec vllm serve "${MODEL_DIR}" \
  --served-model-name "${SERVED_NAME}" \
  --max-model-len "${MAX_MODEL_LEN}" \
  --gpu-memory-utilization "${GPU_MEM_UTIL}" \
  --host 0.0.0.0 \
  --port "${PORT}"

docker build -t hami-vllm-jozu:latest ./vllm
# kind load docker-image hami-vllm-jozu:latest --name <your-cluster>

Step 4: Deploy SGLang Serving the ModelKit

The Deployment uses:

1. initContainer: kitops-init — pulls and flattens the ModelKit into /models/qwen3
2. main container hami-sglang-jozu — serves that local directory
3. HAMi scheduler + gpumem / gpucores caps
4. emptyDir for the model volume (portable; use a PVC in production)

The model layer is about 7.5 GiB. During unpacking, it exists in both KITOPS_HOME and the staging directory, so peak usage is about 15 GiB. Keep the example volume at 20 GiB or larger.

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: kitops
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sglang-modelkit
  namespace: kitops
  labels:
    app.kubernetes.io/name: sglang-modelkit
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: sglang-modelkit
  template:
    metadata:
      labels:
        app.kubernetes.io/name: sglang-modelkit
      annotations:
        hami.io/node-scheduler-policy: binpack
        hami.io/gpu-scheduler-policy: binpack
    spec:
      schedulerName: hami-scheduler
      initContainers:
        - name: kitops-init
          image: hami-kitunpacker:latest
          imagePullPolicy: IfNotPresent
          env:
            - name: MODELKIT_REF
              value: "jozu.ml/jonathangamer202002/qwen3-4b-instruct@sha256:df4629f6a10bba7bec45e12bd15f910ed1024699bfbb44b63240899f71bb1c19"
            - name: UNPACK_PATH
              value: "/models"
            - name: MODEL_SUBDIR
              value: "qwen3"
          resources:
            requests:
              ephemeral-storage: 20Gi
            limits:
              ephemeral-storage: 20Gi
          volumeMounts:
            - name: modelkit
              mountPath: /models
      containers:
        - name: sglang
          image: hami-sglang-jozu:latest
          imagePullPolicy: IfNotPresent
          env:
            - name: MODEL_DIR
              value: "/models/qwen3"
            - name: SERVED_NAME
              value: "qwen3-4b-instruct"
            - name: CONTEXT_LEN
              value: "8192"
            - name: MEM_FRACTION
              value: "0.8"
          ports:
            - name: http
              containerPort: 30000
          resources:
            requests:
              cpu: "2"
              memory: 8Gi
              nvidia.com/gpu: "1"
              nvidia.com/gpumem: "30000"
              nvidia.com/gpucores: "30"
            limits:
              cpu: "8"
              memory: 32Gi
              nvidia.com/gpu: "1"
              nvidia.com/gpumem: "30000"
              nvidia.com/gpucores: "30"
          readinessProbe:
            httpGet:
              path: /health
              port: 30000
            initialDelaySeconds: 40
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 90
          volumeMounts:
            - name: modelkit
              mountPath: /models
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: modelkit
          emptyDir:
            sizeLimit: 20Gi
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 8Gi
---
apiVersion: v1
kind: Service
metadata:
  name: sglang-modelkit
  namespace: kitops
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: sglang-modelkit
  ports:
    - name: http
      port: 8001
      targetPort: http
EOF

For a private registry, add REGISTRY_URL, USERNAME, and PASSWORD env vars to kitops-init (from a Secret). unpack.sh will run kit login before pulling.

Step 5: Watch the ModelKit Unpack

kubectl -n kitops get pods -w
kubectl -n kitops logs -l app.kubernetes.io/name=sglang-modelkit -c kitops-init -f

Successful unpack looks like:

[kitunpacker] ref=jozu.ml/jonathangamer202002/qwen3-4b-instruct@sha256:df4629f6a10bba7bec45e12bd15f910ed1024699bfbb44b63240899f71bb1c19 -> /models/qwen3
[kitunpacker] pulling + unpacking model layers from registry...
Unpacking to /models/.raw-qwen3
...
[kitunpacker] final model directory:
... config.json ... model-00001-of-00003.safetensors ... tokenizer.json ...
[kitunpacker] done.

Then wait for the SGLang container:

kubectl -n kitops rollout status deploy/sglang-modelkit --timeout=30m
kubectl -n kitops logs -l app.kubernetes.io/name=sglang-modelkit -c sglang --tail=50

You should see the custom entrypoint message confirming the model is served from the local ModelKit path:

[sglang-jozu] serving KitOps model from /models/qwen3 (source: Jozu Hub ModelKit)
... model_path='/models/qwen3' ... served_model_name='qwen3-4b-instruct' ...

Step 6: Test Inference

kubectl -n kitops port-forward svc/sglang-modelkit 8001:8001

In another terminal:

curl -s http://127.0.0.1:8001/v1/models | python3 -m json.tool

Example:

{
  "object": "list",
  "data": [
    {
      "id": "qwen3-4b-instruct",
      "object": "model",
      "owned_by": "sglang",
      "max_model_len": 8192
    }
  ]
}

Chat completion:

curl -s http://127.0.0.1:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-4b-instruct",
    "messages": [
      {"role": "user", "content": "In one sentence, what is a KitOps ModelKit?"}
    ],
    "max_tokens": 96,
    "temperature": 0.2
  }' | python3 -m json.tool

If choices[0].message.content is present, ModelKit → local SGLang inference is working.

Step 7: Verify HAMi Caps

POD=$(kubectl get pod -n kitops -l app.kubernetes.io/name=sglang-modelkit -o jsonpath='{.items[0].metadata.name}')
kubectl get pod -n kitops ${POD} \
  -o jsonpath='{.spec.schedulerName}{"\n"}{.spec.containers[0].resources.limits}{"\n"}'
kubectl exec -n kitops ${POD} -c sglang -- env | grep -E 'CUDA_DEVICE|NVIDIA_VISIBLE'
kubectl exec -n kitops ${POD} -c sglang -- nvidia-smi

Verification cluster evidence:

hami-scheduler
... nvidia.com/gpumem:30000 nvidia.com/gpucores:30 ...

NVIDIA_VISIBLE_DEVICES=GPU-...
CUDA_DEVICE_MEMORY_LIMIT_0=30000m
CUDA_DEVICE_SM_LIMIT=30

| NVIDIA H100 80GB HBM3 ... | 24745MiB / 30000MiB |

The main container loaded weights from /models/qwen3 (OCI ModelKit), while HAMi still enforced a 30000 MiB in-pod memory ceiling on the shared H100.

Step 8 (Optional): Co-locate vLLM on the Same ModelKit Pattern

After building/loading hami-vllm-jozu:latest, deploy a second engine with its own HAMi slice. Use a PVC (or node-local cache) if you want both Pods to reuse one unpacked ModelKit; with emptyDir each Pod unpacks independently. Reliable sharing across nodes requires a StorageClass that supports ReadWriteMany. The default kind local-path provisioner is node-local and does not provide a general cross-node shared PVC.

kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-modelkit
  namespace: kitops
  labels:
    app.kubernetes.io/name: vllm-modelkit
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: vllm-modelkit
  template:
    metadata:
      labels:
        app.kubernetes.io/name: vllm-modelkit
      annotations:
        hami.io/node-scheduler-policy: binpack
        hami.io/gpu-scheduler-policy: binpack
    spec:
      schedulerName: hami-scheduler
      initContainers:
        - name: kitops-init
          image: hami-kitunpacker:latest
          imagePullPolicy: IfNotPresent
          env:
            - name: MODELKIT_REF
              value: "jozu.ml/jonathangamer202002/qwen3-4b-instruct@sha256:df4629f6a10bba7bec45e12bd15f910ed1024699bfbb44b63240899f71bb1c19"
            - name: UNPACK_PATH
              value: "/models"
            - name: MODEL_SUBDIR
              value: "qwen3"
          resources:
            requests:
              ephemeral-storage: 20Gi
            limits:
              ephemeral-storage: 20Gi
          volumeMounts:
            - name: modelkit
              mountPath: /models
      containers:
        - name: vllm
          image: hami-vllm-jozu:latest
          imagePullPolicy: IfNotPresent
          env:
            - name: MODEL_DIR
              value: "/models/qwen3"
            - name: SERVED_NAME
              value: "qwen3-4b-instruct"
          ports:
            - name: http
              containerPort: 8000
          resources:
            requests:
              cpu: "2"
              memory: 8Gi
              nvidia.com/gpu: "1"
              nvidia.com/gpumem: "30000"
              nvidia.com/gpucores: "30"
            limits:
              cpu: "8"
              memory: 32Gi
              nvidia.com/gpu: "1"
              nvidia.com/gpumem: "30000"
              nvidia.com/gpucores: "30"
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 40
            periodSeconds: 10
            failureThreshold: 90
          volumeMounts:
            - name: modelkit
              mountPath: /models
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: modelkit
          emptyDir:
            sizeLimit: 20Gi
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 8Gi
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-modelkit
  namespace: kitops
spec:
  selector:
    app.kubernetes.io/name: vllm-modelkit
  ports:
    - name: http
      port: 8000
      targetPort: http
EOF

Test with:

kubectl -n kitops port-forward svc/vllm-modelkit 8000:8000
curl -s http://127.0.0.1:8000/v1/models

Ensure combined gpumem requests fit on the physical GPU. Two requests of 30000 MiB require 60000 MiB, or about 58.6 GiB, free on that card.

Reference: Kitfile (repack your own ModelKit)

# Reference Kitfile for the Qwen3-4B-Instruct ModelKit used in this demo.
#
# The demo PULLS a pre-built public ModelKit from Jozu Hub:
#     jozu.ml/jonathangamer202002/qwen3-4b-instruct@sha256:df4629f6a10bba7bec45e12bd15f910ed1024699bfbb44b63240899f71bb1c19
#
# This Kitfile is provided so you can (re)pack and push your OWN ModelKit to a
# registry (Jozu Hub, ACR, GHCR, ...) from a local model directory (config.json + safetensors):
#
#     # 1) get a model directory (e.g. via `kit unpack` or `kit import`)
#     # 2) place this Kitfile next to a ./qwen3 directory of safetensors + config
#     kit pack . -t jozu.ml/<your-org>/qwen3-4b-instruct:latest
#     kit login jozu.ml -u <user> --password-stdin   # needed only for push
#     kit push jozu.ml/<your-org>/qwen3-4b-instruct:latest
manifestVersion: "1.0"
package:
  name: qwen3-4b-instruct
  version: "1.0"
  authors:
    - HAMi KubeCon Demo
  description: >
    Qwen3-4B-Instruct-2507 packaged as a KitOps ModelKit (safetensors layout), served on HAMi-virtualized GPUs by vLLM and SGLang.

model:
  name: qwen3-4b-instruct
  path: ./qwen3
  license: Apache-2.0
  description: Qwen3 4B instruct, safetensors (Qwen3ForCausalLM)

# After placing a safetensors-layout model directory at ./qwen3 next to the Kitfile:
kit pack . -t jozu.ml/<your-org>/qwen3-4b-instruct:latest
kit login jozu.ml
kit push jozu.ml/<your-org>/qwen3-4b-instruct:latest

Then point MODELKIT_REF in the Deployment at your tag.

Troubleshooting

Symptom	What to Check
initContainer stuck pulling	Registry reachability from the node; disk pressure on emptyDir; increase sizeLimit.
config.json not found after unpack	ModelKit layout differs; inspect with kit inspect --remote and adjust flatten logic / MODEL_SUBDIR.
SGLang exits: model dir missing	initContainer failed; kubectl logs ... -c kitops-init.
ImagePullBackOff for custom images	kind load / push to your registry; set imagePullPolicy: IfNotPresent for local tags.
Pod Pending on GPU	Free HAMi shares; lower gpumem; confirm hami-scheduler events.
Private registry 401	Set REGISTRY_URL / USERNAME / PASSWORD on kitops-init.
In-pod memory still full GPU size	Verify the HAMi environment variables and schedulerName.

Cleanup

kubectl delete namespace kitops --ignore-not-found
# optional: remove local images
# docker rmi hami-kitunpacker:latest hami-sglang-jozu:latest hami-vllm-jozu:latest

Verification Results

Claim	Evidence
Model is an OCI ModelKit	kit inspect --remote returns KitOps manifest / model layers.
Model delivered from the ModelKit into the main container	initContainer logs show kit unpack; SGLang logs show serving KitOps model from /models/qwen3 and model_path='/models/qwen3'.
HAMi schedules the workload	schedulerName: hami-scheduler + Filtering/Binding events.
GPU memory/compute caps apply	CUDA_DEVICE_MEMORY_LIMIT_0=30000m, CUDA_DEVICE_SM_LIMIT=30; in-pod nvidia-smi shows ... / 30000MiB.
Inference works	/v1/models lists qwen3-4b-instruct; chat completions return content.

Next Steps

• Swap the public Jozu ModelKit for your internal registry ModelKit and wire imagePullSecrets / kit login Secrets.
• Share one ReadWriteMany PVC across SGLang and vLLM so the ModelKit is unpacked once across nodes.
• Combine with Lab 3: GPU Partitioning to pack more tenants per GPU.
• For a simpler debugging path, run SGLang with a model pulled directly at startup before adding the ModelKit supply-chain workflow.