# Proposed Ray Architecture for CheckMAITE

# Summary

CheckMAITE must support three related but different workload paths:

1. Models that remain available for inference, regardless of whether anyone has a notebook open.
2. Interactive capability executions submitted from notebooks, often using datasets, models, or Python objects already created in notebook memory.
3. Capability executions submitted through the CheckMAITE UI or REST API. These use durable request data and must continue after the HTTP request ends.

Notebook and UI/API submissions run the same CheckMAITE capabilities and receive the same user-facing lifecycle, status, cancellation, result, and reporting standards. They use different submission mechanisms because their inputs differ: notebooks can hold live Python objects, while UI/API requests use durable catalog specifications and data references.

Running model serving and all capability execution on one shared, always-running Ray cluster is convenient, but it couples unrelated availability, security, scaling, and software-version concerns. A large execution can consume resources needed for inference. A serving upgrade can replace the cluster while other work is running. Anyone with Ray Client access joins the same trust domain. Keeping a live Ray Client endpoint available also prevents the entire cluster from scaling to zero.

The proposed design separates these responsibilities and uses the submission mechanism appropriate to each client:

• A dedicated RayService keeps shared models available and protects their capacity from capability executions.
• Each trusted group receives a Kubernetes namespace with its own quota, RBAC, network controls, storage access, and Ray resources.
• Notebooks use Ray Client against an optional warm group RayCluster so they can submit live Python objects with low latency.
• The UI and REST API create a KubeRay RayJob in the caller’s group namespace using durable, declarative inputs. The RayJob can use a suitable warm group cluster or create an ephemeral cluster that is removed after completion.
• Groups that only use UI/API submission can have no Ray pods while idle. Groups that need Ray Client keep a small head available, while their compute workers scale to zero.
• Notebook jobs remain discoverable through the CheckMAITE Ray registry, while UI/API runs are recorded in the API database and linked to their RayJob CRs. The two paths do not require a shared execution ID or cross-channel job management.

This hybrid design treats notebooks and UI/API as equally important workflows without forcing either one into the other’s submission model.

# 1. Requirements

# User requirements

• Users can submit CheckMAITE capabilities directly from Python notebooks.
• Users can submit CheckMAITE capabilities through an authenticated UI and REST API.
• Users can pass serializable capability, dataset, model, and configuration objects they have already created in a notebook.
• UI/API requests use durable catalog specifications and references to datasets, models, and configuration that the job can load independently.
• Submitting work is fast enough to feel interactive, and an API request returns an API run ID without waiting for completion.
• Closing a notebook or ending an HTTP request does not automatically stop work that has already been accepted.
• Notebook users can list, inspect, cancel, or retrieve notebook-submitted jobs through the notebook job API.
• UI/API users can list, inspect, cancel, or retrieve UI/API-submitted runs through the UI or REST API.
• Notebook and UI/API users are not required to discover or manage jobs submitted through the other path.
• Common models remain loaded and ready for inference.
• Capabilities such as NRTK can repeatedly request predictions for generated or perturbed data.
• User groups have separate capacity and cannot access each other’s Ray environments.
• Results, reports, analytics, and provenance are stored outside short-lived worker processes.

# Operational requirements

• Notebook and UI/API capability executions should not slow down or interrupt shared model inference.
• Each group has clear Kubernetes-enforced limits on how much CPU, memory, GPU, and storage capacity it can use.
• Operators can identify which resources belong to a particular CheckMAITE execution.
• Updating model-serving infrastructure does not terminate unrelated CheckMAITE work.
• A notebook crash or disconnected UI/API client does not remove the only way to find or cancel a running execution.
• Finished and abandoned Ray actors, RayJobs, and clusters are cleaned up automatically, either immediately or after a suitable retention timeout.
• Users receive a clear distinction between work waiting for resources and work that is actually running.

# Priorities and scope

• Notebook and UI/REST submissions are equally important supported workflows. They use separate job records, identifiers, and management interfaces, but follow consistent status, cancellation, result, reporting, and reliability standards.
• Isolation between trusted groups is a hard requirement. Members of one trusted group are treated as mutually trusted in the current design. Stronger isolation between members of the same group is a possible future improvement, not a requirement for the initial architecture.
• Reliability is a core requirement. Model availability, reliable handling of accepted executions, accurate status, cancellation, result storage, recovery, and cleanup take priority over optional convenience, optimization, and user-interface features.

# 2. Ray and CheckMAITE terminology

The word job is overloaded:

• A CheckMAITE job, called a CheckMAITE execution in this document, is one submitted capability run. It has a CheckMAITE ID, status, result, report, and analytics records.
• A Ray task is the remote Python function that performs some or all of that execution.
• A Ray actor is a long-running process supporting the execution, such as its controller or model.
• A Ray driver job is a connected Python process, such as a notebook kernel or the entrypoint started for a RayJob. Several CheckMAITE executions may appear under one notebook driver because they came through the same connection. A driver does not reserve an entire Ray node.
• A RayJob is the KubeRay custom resource proposed for UI/API submissions. It is different from Ray Client and from calling Ray’s Jobs HTTP API directly.

These identifiers should not be treated as interchangeable. A notebook job ID is canonical within the notebook path, while the API run ID is canonical within the UI/API path. A global cross-channel ID is not required. Ray and Kubernetes identifiers remain supporting diagnostic information.

# 3. High-level architecture

flowchart TB
    subgraph Clients["CheckMAITE clients"]
        NA["Notebook — Group A"]
        NB["Notebook — Group B"]
        UI["UI / REST clients"]
    end

    subgraph Services["UI/API execution services"]
        SUBMIT["Authenticated submission API"]
        MANAGE["UI/API status and cancellation API"]
        DB[("PostgreSQL run records")]
        STORE[("Datasets, reports, analytics, and results")]
        SUBMIT --- DB
        MANAGE --- DB
    end

    KAPI["Kubernetes API / KubeRay operator"]

    subgraph GroupA["Group A namespace · quota · RBAC"]
        RCA["Warm RayCluster A\nnotebook registry · workers 0→N"]
        RJA["RayJob CRs"]
        EPA["Ephemeral job clusters"]
        RJA -. "low-latency clusterSelector" .-> RCA
        RJA -->|"isolated run"| EPA
    end

    subgraph GroupB["Group B namespace · quota · RBAC"]
        RCB["Warm RayCluster B\nnotebook registry · workers 0→N"]
        RJB["RayJob CRs"]
        EPB["Ephemeral job clusters"]
        RJB -. "low-latency clusterSelector" .-> RCB
        RJB -->|"isolated run"| EPB
    end

    subgraph Serving["Shared model-serving environment"]
        GW["Authenticated inference endpoint"]
        RS["Dedicated RayService"]
        MODELS["Always-on shared models"]
        GW --> RS --> MODELS
    end

    NA -->|"Ray Client · live objects"| RCA
    NB -->|"Ray Client · live objects"| RCB
    UI -->|"durable specification"| SUBMIT
    SUBMIT --> KAPI
    KAPI -->|"authorized Group A create"| RJA
    KAPI -->|"authorized Group B create"| RJB
    UI --> MANAGE
    RCA --> STORE
    RCB --> STORE
    EPA --> STORE
    EPB --> STORE
    RCA -->|"shared-model inference"| GW
    RCB -->|"shared-model inference"| GW
    EPA -->|"shared-model inference"| GW
    EPB -->|"shared-model inference"| GW

Notebook kernels connect only to their trusted group’s warm RayCluster. UI/REST clients never receive Ray Client credentials: the submission API authenticates the caller, records a UI/API run in PostgreSQL, creates a RayJob in the authorized group namespace, and returns the API run ID without waiting for the capability to finish.

A RayJob may select a warm group RayCluster for lower startup latency, or create an ephemeral cluster for stronger job isolation, environment pinning, or specialized CPU/GPU resources. Group quota and RBAC apply in either case. The serving environment remains a separate RayService with independently protected capacity.

# 4. Shared model-serving environment

# Purpose

This environment hosts models that should remain available even when no notebook or CheckMAITE execution is active. Examples include an approved reference model used by several evaluations or a model exposed to other applications for online inference.

A dedicated KubeRay RayService is appropriate because it manages model replicas, monitors their health, scales them in response to demand, and supports updates without intentionally interrupting inference traffic.

# Access and authorization

Users do not connect their notebooks directly to the serving Ray cluster. They access only a controlled inference endpoint.

CheckMAITE computation
  → authenticated inference request
  → serving proxy
  → approved model
  → prediction response

The inference endpoint identifies the calling user or group and can enforce:

• which models the group may use;
• request and concurrency limits;
• audit logging;
• model-version selection;
• payload-size limits;
• access to sensitive models.

For example:

Group A
  - can call models reference-model-v1 and detector-v3
  - cannot view the serving dashboard
  - cannot deploy, replace, or delete a model
  - cannot access Group B's restricted model

Group B
  - can call reference-model-v1 and group-b-model-v2
  - cannot connect to Group A's compute cluster
  - cannot access the serving cluster's Ray Client or internal control ports

Serving administrators
  - can deploy and update approved models
  - can inspect serving health and capacity

Authentication should occur at an API gateway or internal service endpoint. Direct access to the serving cluster’s Ray Client, dashboard API, and internal control services should be blocked from notebook environments.

# Illustrative capacity

Capacity depends more on model cost, latency, batching, and simultaneous requests than on the total number of registered users. Ten users making occasional requests may need less capacity than one NRTK execution sending hundreds of images at once.

An illustrative CPU serving configuration might be:

Head/control pod:
  1–2 Kubernetes CPUs and 4–8 GiB memory
  0 Ray CPUs available for model or user work

Serving workers:
  minimum 2 workers, each 2 CPUs and 4–8 GiB
  enough room for at least 2 model replicas

Model replicas:
  minimum 2 replicas for availability
  1 CPU each for a lightweight CPU model
  autoscale to 4–8 replicas after load testing

For a GPU model, the equivalent baseline might be two replicas on two GPU workers so that one pod or node failure does not remove all serving capacity.

These are examples, not final sizing recommendations. Representative inference tests should determine replica CPU/GPU requirements, maximum concurrent requests, and useful batch sizes.

A RayService update can temporarily run both the old and replacement clusters. Kubernetes must therefore have enough spare capacity—or autoscaling headroom—to host roughly twice the normal serving footprint during an upgrade.

# Relationship to the current deployment

The current COSMOS development deployment does not separate serving and CheckMAITE compute:

• one RayService supplies both the Serve application and general Ray Client access;
• the CheckMAITE API points at that existing Ray Client endpoint, so a live head is required at submission time;
• direct Ray Client access has no application-level user authentication and transfers pickled Python code and objects;
• the API, notebook, and cluster environments must remain closely version-compatible;
• the head advertises CPUs for user work;
• the healthz model replica runs on the head;
• CheckMAITE tasks may run on either the head or worker;
• both Ray pods currently run on the same physical Kubernetes node;
• the active cluster has one worker despite the Git configuration requesting four;
• serving and CheckMAITE work therefore compete for the same four logical Ray CPUs.

The proposed design replaces that shared pool with dedicated serving capacity, optional warm notebook clusters, and group-scoped RayJobs. Kubernetes placement and quota should ensure serving replicas use serving capacity while notebook and UI/API computations use only their authorized group resources.

# Model versioning

Each CheckMAITE execution should record the exact model revision it used. A model update must not cause one execution to receive predictions from two different revisions.

Versioned endpoints could look conceptually like:

/models/detector/v3/predict
/models/detector/v4/predict

An execution remains pinned to one version even if a newer model becomes available during the run.

# 5. Group CheckMAITE compute environments

# One group-scoped stack per trusted group

Each trusted group receives a Kubernetes namespace containing its quota, RBAC, NetworkPolicy, service accounts, storage mounts, RayJobs, and—when the group needs notebook access—an optional persistent RayCluster.

Group A → checkmaite-group-a namespace → optional warm RayCluster + RayJobs
Group B → checkmaite-group-b namespace → optional warm RayCluster + RayJobs

The namespace and policy remain even when no Ray pods are running. Members connected to the same warm RayCluster are mutually trusted at the Ray level. If users within a group must not be able to interfere with each other, use separate clusters or an ephemeral RayJob cluster for the sensitive execution.

# Notebook path: Ray Client

Notebook code connects to its group’s warm RayCluster through Ray Client and can submit live Python objects. This preserves the current low-latency workflow and the existing CheckMAITE registry/controller lifecycle.

A notebook group therefore keeps a small head available while ordinary CPU/GPU workers scale from zero to the group’s limit. The head advertises zero Ray CPUs for ordinary capability work. Multiple notebook drivers can share the same head and workers; a driver does not reserve a whole node.

Ray Client requires close Python, Ray, CheckMAITE, and dependency compatibility between the notebook kernel and group cluster. A lost head also ends Ray Client sessions and loses inputs that existed only in notebook memory.

# UI/API path: RayJob

The UI and REST API submit a declarative CheckMAITE request through an authenticated service. The request contains catalog specifications and durable data references rather than pickled notebook objects. The service:

1. authenticates the caller and resolves the trusted group;
2. creates the UI/API run record in PostgreSQL;
3. creates a RayJob in that group’s namespace using group-scoped Kubernetes credentials;
4. returns the API run ID immediately;
5. mirrors RayJob state, logs, and result locations into the API run record.

The RayJob entrypoint loads the requested CheckMAITE capability, model, dataset, and configuration from durable references. It should run from a versioned image; a pinned runtime environment may be useful during development, but production should not resolve large dependency sets on every run.

A RayJob has two placement options:

• Selected warm cluster: use clusterSelector to run on the group’s existing RayCluster when low startup latency and shared group capacity are appropriate.
• Ephemeral job cluster: let KubeRay create a cluster for the job when stronger isolation, a different software environment, a GPU shape, or independent teardown is more important.

UI/API clients never receive Ray Client credentials or direct Kubernetes access. The submission service is trusted platform infrastructure. It must authorize the group before creating anything, use narrowly scoped credentials, and prevent a caller from choosing another group’s namespace, service account, storage, or cluster selector.

# Warm and dormant groups

A group can choose between two idle modes:

• Dormant: no RayCluster and no Ray pods. UI/API RayJobs create ephemeral capacity on demand. This reaches true Ray scale-to-zero but includes cluster, pod, image, and possibly node startup latency.
• Warm: a small Ray head remains available for Ray Client and latency-sensitive RayJobs. Compute workers still scale to zero.

A group that requires interactive notebook submission cannot be completely dormant because Ray Client needs a live head at connection and submission time. Keeping every group warm would multiply head memory and CPU cost, so dormant should be the default for UI/API-only groups and warm status should be enabled only where its latency benefit is needed.

# Illustrative layout and sizing

A warm group environment starts conceptually with:

Head pod:
  small but measured Kubernetes CPU and memory request
  0 Ray CPUs for ordinary capability computations

CPU workers:
  minimum 0
  autoscale to the group's quota

GPU workers:
  minimum 0
  autoscale only for declared GPU work

Ephemeral RayJob clusters:
  no idle pods
  size and software image declared per approved job profile

Final pod requests, object-store memory, maximum workers, and warm-head size must come from ATEP dev measurements rather than assuming chart defaults. Node autoscaling and image pulls can make a fully dormant cold start substantially slower than a warm-cluster submission.

# Ray Client control actors

The persistent registry and controller actors apply to the notebook/Ray Client path:

Registry actor:
  explicitly 0 Ray CPUs

Controller actor:
  approximately 0.01 Ray CPU per active execution

Typical capability worker:
  1 Ray CPU unless overridden

A typical notebook execution therefore needs approximately 1.01 logical CPUs while its worker is active. Control actors must run on the head or fixed control capacity so they do not keep autoscaled compute workers alive. Terminal actors should be removed immediately or after the configured retention timeout.

RayJob executions use the RayJob and its driver as the infrastructure lifecycle record rather than requiring detached CheckMAITE registry/controller actors to keep the cluster alive. PostgreSQL remains the durable UI/API run record after RayJob and cluster cleanup. It does not contain or reconcile notebook registry jobs.

# Shared storage is required

Both paths need durable storage visible to the submitting service and the selected or ephemeral Ray workers. Notebook-local paths are not sufficient. Group-scoped datasets, reports, analytics, and results must use the shared filesystem or another durable artifact store before dormant RayJob execution can be reliable.

# 6. CheckMAITE execution lifecycle

The two submission paths keep independent lifecycle records. They should use consistent plain-language states, but they do not share a job database, identifier, listing, or cancellation interface. Notebook users see notebook registry jobs; UI/API users see PostgreSQL-backed runs and their RayJob status.

# Notebook lifecycle

The notebook path continues to use the group registry and execution controller. The controller accepts responsibility before submission returns success, starts the capability worker, and publishes status and durable result locations. A replacement notebook can reconnect to the same group cluster and stable scope:

configure_job_backend(
    "ray",
    address=group_cluster_address,
    idempotency_scope=stable_workspace_scope,
)

jobs = list_jobs()

Each warm group cluster has one default registry. Stable user/workspace scopes organize and deduplicate records inside it. Losing the whole head loses the registry and arbitrary inputs that existed only in notebook memory. Previously persisted analytics or artifacts may survive, but no UI/API run record is expected to reconstruct or list the lost notebook jobs.

# UI/API RayJob lifecycle

sequenceDiagram
    participant U as UI or REST client
    participant A as CheckMAITE API
    participant D as PostgreSQL run database
    participant K as Kubernetes API
    participant O as KubeRay operator
    participant J as RayJob driver
    participant S as Durable artifact store

    U->>A: Submit catalog specification and data references
    A->>D: Create UI/API run record and ID
    A->>K: Create RayJob in authorized group namespace
    K-->>A: RayJob accepted
    A-->>U: Return API run ID
    O->>K: Observe RayJob
    O->>J: Select or create cluster and start entrypoint
    A->>D: Mirror provisioning and running status
    J->>S: Read inputs and write reports, analytics, and results
    J-->>K: Publish terminal RayJob status
    A->>D: Record terminal state and artifact locations
    O->>K: Remove ephemeral cluster after retention period
    U->>A: Inspect, cancel, or retrieve by API run ID
    A-->>U: Return UI/API run response

The RayJob CR is a useful Kubernetes lifecycle record and survives an ephemeral cluster teardown while it is retained. It is not the long-term UI/API database: Kubernetes retention settings may delete it, while PostgreSQL and artifact storage retain the run history and results. This database does not need to contain notebook jobs.

# Required behavior

1. A notebook submission receives a notebook job ID from the Ray registry; a UI/API submission receives an API run ID from PostgreSQL.
2. Neither interface is required to list, inspect, cancel, or retrieve jobs submitted through the other interface.
3. UI/API submission chooses the namespace from authenticated group membership, never from an untrusted request field.
4. Idempotent UI/API submission maps the API run to a deterministic RayJob name or label and does not create duplicate work after a retry.
5. Submission reports success only after the registry/controller has accepted notebook work or Kubernetes has accepted the RayJob.
6. Each interface distinguishes cluster provisioning or resource scheduling from capability execution where those phases apply.
7. Provisioning/scheduling timeouts are separate from execution timeouts.
8. Notebook completion updates the Ray registry and durable artifacts. UI/API completion updates PostgreSQL and durable artifacts.
9. Cancellation independently stops the notebook controller or UI/API RayJob and cleans up its owned infrastructure after the configured retention period.
10. Actor and task names include the notebook job ID; RayJob, pod, and cluster labels include the API run ID.
11. UI/API reconciliation compares PostgreSQL only with RayJobs. Notebook registry cleanup and recovery remain internal to the Ray Client backend.
12. Terminal actors, RayJobs, and ephemeral clusters are cleaned up automatically, either immediately or after a suitable timeout.

# Client disconnection

After either path accepts the work, losing a notebook or ending an HTTP request does not by itself stop the execution. Notebook users reconnect through the stable group registry using notebook job APIs. UI/API users reconnect through the UI/API using the API run ID. Neither reconnection path queries the other path’s records.

# 7. Model execution patterns

# Shared approved model

Use the dedicated serving environment:

NRTK computation
  → generate perturbed image batch
  → send inference request
  → shared versioned model
  → receive predictions

NRTK can generate many model requests, so requests should be batched and concurrency should be bounded. Sending every image as an individual base64-encoded JSON request is likely to create unnecessary network and serialization overhead.

# Model supplied from a notebook

A model created or loaded in a notebook can be sent through the interactive Ray connection and loaded into a model actor in that group’s cluster.

Notebook model object
  → group RayCluster
  → long-running model process
  → CheckMAITE computation requests predictions

This avoids crossing from the group cluster to the shared serving cluster and preserves the convenience of passing live Python objects.

A full Ray Serve deployment is unnecessary when only one CheckMAITE execution needs the model. A model actor is simply a stateful Python process that keeps the model loaded for repeated calls.

# Model referenced by a UI/API submission

A UI/API request cannot contain a live model object. It identifies an approved catalog model, durable model artifact, or shared serving endpoint. The RayJob entrypoint resolves that immutable reference and either loads the model in its selected cluster or calls the authenticated shared inference endpoint. The UI/API run record stores the exact model revision.

# Model reused by a group

A model used by several notebook or selected-cluster executions in one trusted group can remain in a named group-owned actor on the warm cluster. It should have:

• an immutable model/version identity;
• a declared CPU or GPU requirement;
• ownership information;
• a lease or reference count;
• an idle timeout;
• explicit cleanup behavior.

# Group-specific model service

If a group needs its own HTTP endpoint, independently scalable replicas, or request batching shared across many executions, deploy a separate RayService for that group. Arbitrary user models should not be deployed into the globally shared serving environment.

# 8. Isolation and authorization

Ray does not provide strong user isolation inside one Ray cluster. A notebook user who can connect through Ray Client can execute arbitrary Python in that warm group cluster, so everyone with that access is part of one trusted group. UI/API callers do not receive Ray Client access; Kubernetes namespaces, RBAC, quotas, and the selected or ephemeral Ray cluster enforce their group boundary.

# Is a Kubernetes namespace a security boundary?

A namespace is not a security boundary by itself. By default, workloads in different namespaces may still communicate, and permissions may still be granted too broadly.

A namespace becomes part of an effective isolation boundary only when it is combined with:

• Kubernetes RBAC limiting which users and service accounts can access resources in it;
• NetworkPolicy blocking traffic from other namespaces;
• a dedicated service account with minimal permissions;
• separate Ray credentials;
• ResourceQuota and LimitRange;
• controlled access to storage credentials and object-store prefixes;
• authenticated dashboard and management routes.

The isolation boundary in this proposal is therefore:

group-scoped namespace
+ group warm and/or ephemeral Ray clusters
+ RBAC and admission rules for RayJob creation
+ NetworkPolicy
+ separate service accounts and credentials
+ resource and storage limits

A Ray namespace is different from a Kubernetes namespace. A Ray namespace only organizes names and actors inside one Ray cluster; it does not enforce network access, permissions, or resource quotas.

# Serving isolation

The serving environment exposes only approved inference endpoints to user groups. Notebook and UI/API users cannot directly access its Ray Client, internal cluster control service, or deployment administration APIs.

Each inference request carries a user or group identity. The gateway can then enforce model permissions, rate limits, audit records, and group-specific routes.

# Execution ownership and visibility

Notebook jobs are recorded in the CheckMAITE Ray registry and exposed through notebook list_jobs(), get_job(), cancellation, and result APIs. UI/API runs are recorded separately in PostgreSQL with their owner, group, RayJob identifiers, status, runtime location, durable input references, and result locations.

There is no required user-facing query across the two stores. The UI/API does not list notebook registry jobs, and notebook job APIs do not list RayJobs. Operators can correlate infrastructure through group, namespace, and path-specific execution labels when troubleshooting.

The submission service must not use one unrestricted credential that can create arbitrary resources in every namespace. Group authorization must be enforced when selecting a namespace and backed by narrowly scoped service accounts, RBAC, admission policy, or separate group-scoped submission components. A compromised submission path must not be able to choose another group’s storage, service account, or warm cluster.

Members with Ray Client access to one warm group cluster remain mutually trusted. Per-user clusters are not a requirement for the initial notebook design, but an ephemeral RayJob cluster can provide stronger isolation for an individual UI/API execution when needed.

# 9. Scaling and scheduling

# Shared serving environment

Scaling occurs in layers:

More inference requests
  → start more model replicas
  → start more Ray worker pods if needed
  → start more EKS nodes if the pods do not fit

An illustrative starting point for a lightweight CPU model is:

2 model replicas minimum
1 CPU per replica
2 serving workers minimum
4–8 replicas maximum after load testing

A service that must survive one worker failure should not place all minimum replicas on one pod or physical node. GPU models need the same consideration at the GPU-node level.

Capacity should be measured in concurrent inference requests and model throughput, not simply number of users. Dynamic batching may allow one replica to serve many users efficiently.

# Group compute environments

Notebook and UI/API paths scale differently:

Notebook submission
  → existing group head accepts work
  → Ray scales workers 0→N
  → EKS adds nodes if required

UI/API submission to dormant group
  → RayJob is created
  → KubeRay creates an ephemeral cluster
  → EKS adds nodes if required
  → cluster is removed after completion

A warm group pays only for its head while no computations are active; its ordinary CPU and GPU workers have a minimum of zero. A dormant group has no Ray pod cost while idle. The practical cost and latency still depend on Kubernetes system services, available node capacity, image caching, storage attachment, and EKS node autoscaling.

The key settings are:

• whether the group is dormant or has a warm Ray Client head;
• desired number of simultaneous CheckMAITE executions;
• approved RayJob profiles and whether they select a warm or ephemeral cluster;
• CPU, GPU, memory, and object-store requirements per execution;
• maximum workers and total group quota;
• acceptable queue and cold-start time;
• acceptable idle and burst cost.

Kubernetes ResourceQuota limits the total pods and resources created in each group namespace, including overlapping warm-cluster and ephemeral RayJob work. Ray schedules tasks within a cluster. CheckMAITE still needs per-user concurrency and pending limits; an optional queueing system can be added later if measured contention requires cross-group fairness.

# Cost interpretation

“Scale to zero” must identify what reaches zero. A dormant group has no Ray pods, but shared Kubernetes, KubeRay, ingress, monitoring, database, and storage services still run. A warm notebook group retains a Ray head but can have zero compute workers. Actual AWS savings occur only when the resulting free pod capacity allows the EKS node autoscaler to remove a node.

This makes the Ray/Dask comparison a deployment-policy comparison rather than an inherent scheduler limitation: an on-demand RayJob cluster and an on-demand Dask cluster can both have no per-group compute pods while idle. Ray Client deliberately trades a small warm-head cost for live-object submission and lower latency.

# Admission and backpressure

CheckMAITE should enforce:

• notebook concurrency limits within the Ray Client backend and UI/API concurrency limits within the API backend, without requiring a combined per-user count;
• path-specific maximum queued, provisioning, or scheduling executions, with Kubernetes quota providing the shared group-wide ceiling;
• an allowlist of RayJob images, entrypoints, service accounts, and resource profiles;
• validation of CPU, memory, GPU, namespace, cluster-selector, and storage requests;
• separate timeouts for cluster provisioning, resource scheduling, and capability execution;
• limits on simultaneous requests to shared models;
• clear explanations when a request is blocked by quota or cannot fit on any configured worker or node type.

# 10. Failure and upgrade behavior

Keeping serving, warm notebook clusters, and ephemeral RayJob clusters separate reduces upgrade coupling. Reliability still depends on explicit reconciliation, bounded retries, durable artifacts, and honest status transitions; KubeRay lifecycle automation does not replace those product responsibilities.

# 11. Visibility and operations

Within either path, users should not need to understand several unrelated Ray or Kubernetes IDs to answer basic questions such as:

• Is my job waiting or running?
• Which model version is it using?
• Why has it not started?
• Can I cancel it after my notebook crashed or browser closed?
• Where are its results and reports?

The notebook job API presents a notebook job ID and registry status. Ray actor, task, driver, pod, and node identifiers remain notebook-path diagnostics. Separately, the UI/API presents an API run ID and PostgreSQL-backed status; RayJob, cluster, pod, and driver identifiers remain UI/API diagnostics.

The UI/API management view should show:

• owner and group;
• current state and how long it has been in submission, provisioning, scheduling, or execution;
• warm-cluster or ephemeral-cluster placement;
• requested resources, namespace quota, and whether suitable capacity is available;
• model, CheckMAITE, and job-image versions;
• result and report location;
• clear completion, cancellation, timeout, or failure reason.

Notebook users continue using the notebook job handles, list_jobs(), and get_job() rather than the UI/API management view. No combined user-facing list is required.

Operators still need infrastructure views across both paths showing group quota, warm-head and worker capacity, queued work, stale controllers or RayJobs, serving saturation, and clusters or pods that remain after executions finish. Alerts should focus on user-impacting conditions rather than raw actor counts alone.

# 12. Hybrid submission design

The architecture deliberately supports two independent submission systems that share platform infrastructure. The notebook library uses the Ray Client backend, while the CheckMAITE UI/API uses the RayJob backend. Neither system fronts, lists, or controls the other’s jobs.

# Why notebooks retain Ray Client

Ray Client can transfer functions and serializable objects already present in notebook memory. A RayJob starts a command from a declarative specification and cannot directly capture that live notebook state. Forcing notebooks through RayJob would require staging code and inputs first and would remove the low-latency interactive behavior users require.

# Why UI/API uses RayJob

The frontend and REST API already operate on serializable catalog specifications rather than live Python objects. A RayJob gives that path a Kubernetes-visible submission and lifecycle record, group namespace enforcement, resource quota, a pinned execution environment, and optional ephemeral-cluster teardown. It also removes the need for the API process and job image to share one cloudpickle-by-reference environment.

For UI/API submissions, the public contract remains the CheckMAITE API and API run ID. The RayJob name, Kubernetes conditions, driver job, cluster, and pods are implementation and diagnostic details. A versioned job image is preferred; development may use a pinned runtime environment, but arbitrary per-job package installation increases startup time and weakens reproducibility.

# Lifecycle and cleanup

The UI/API backend should label each RayJob with its API run ID, owner, group, and immutable version information. An idempotent request maps to one RayJob. KubeRay may shut down an owned cluster after completion, and Kubernetes may retain or delete the RayJob CR after a configured timeout. Before cleanup, the API must capture terminal status, logs or log location, and durable result references in PostgreSQL.

UI/API cancellation requests go through the CheckMAITE API. Its backend stops or deletes the RayJob and owned cluster in a controlled order, records cancellation in PostgreSQL, and retains user artifacts. Direct deletion of a CR is not the user-facing lifecycle contract.

# POC evidence and production qualification

The POC demonstrates the important feasibility points: fast RayJob CR creation, on-demand cluster materialization, worker scale-to-zero, concurrent group namespaces, Kubernetes RBAC denial across groups, ephemeral-cluster teardown, and queryable RayJob state after cluster removal. That supports adopting the hybrid design.

POC startup measurements are not a production latency guarantee. Production validation must include cold image pulls, EKS node scale-up, storage mounts, network policy, real CheckMAITE images, failures, retries, upgrades, and concurrent quota pressure. A fully dormant request may take about a minute or longer from submission to capability start; warm-cluster selection is the latency-control option.

# 13. Tradeoffs

# Benefits

• Preserves direct, low-latency Python-object submission from notebooks.
• Gives UI/REST work a declarative, independently versioned RayJob path.
• Lets each path keep a simple, purpose-built job store and management interface while following consistent lifecycle and reliability standards.
• Allows UI/API-only groups and ephemeral job clusters to have zero Ray pods while idle.
• Keeps shared models available independently of notebook and UI/API activity.
• Prevents routine serving upgrades from terminating unrelated capability executions.
• Provides meaningful isolation and independent quotas between trusted groups.
• Allows latency-sensitive UI/API work to select warm capacity and larger or sensitive work to use an ephemeral cluster.
• Supports shared approved models, durable catalog models, and arbitrary notebook-provided models.
• Allows users to reconnect after losing a notebook or UI/API connection.

# Costs and limitations

• CheckMAITE must operate two independent lifecycle implementations: registry/controllers for Ray Client and PostgreSQL/RayJob reconciliation for UI/API. They do not reconcile with each other.
• Groups that need notebook Ray Client access retain a small always-running head cost even when workers scale to zero.
• Dormant RayJob execution adds cluster, pod, image, storage, and possibly EKS node startup latency.
• Members with Ray Client access to one warm group cluster remain mutually trusted within that cluster.
• Passing arbitrary notebook objects requires compatible Python, Ray, CheckMAITE, and dependency environments.
• UI/API jobs require durable specifications, accessible datasets, versioned images, and durable artifact storage.
• Objects containing notebook-local file paths still require equivalent worker mounts or durable storage.
• Calls from a group or ephemeral cluster to a shared model require network serialization and careful batching.
• Losing a warm RayCluster cannot reconstruct inputs that existed only in notebook memory.
• Kubernetes RayJob records are operational state, not a substitute for PostgreSQL UI/API run history and retained results.
• ResourceQuota contains the combined group infrastructure but does not by itself provide fair queueing. Each submission path enforces its own per-user limits; additional queueing should be added only when required.

# 14. Key decisions

1. Use a dedicated RayService for managed, always-on model serving.
2. Give each trusted group a Kubernetes namespace with its own quota, RBAC, network policy, credentials, and storage access.
3. Retain Ray Client for notebooks so they can submit live Python objects with low latency.
4. Provide an optional warm RayCluster for groups that need notebook or low-latency access; keep its head free of user work and scale ordinary workers from zero.
5. Use KubeRay RayJob CRs for UI and REST API submissions based on durable specifications.
6. Use ephemeral RayJob clusters by default for dormant, specialized, or strongly isolated work, with approved warm-cluster selection when latency matters.
7. Retain and harden CheckMAITE’s registry/controller model for the Ray Client path, without requiring detached actors for the RayJob lifecycle.
8. Keep notebook job identity and management in the Ray registry; keep UI/API run identity and management in PostgreSQL plus RayJob CRs. Do not require cross-channel discovery or control.
9. Use job-specific model actors for live models supplied from notebooks and immutable durable references for UI/API models.
10. Use authenticated inference endpoints for shared approved models.
11. Use versioned job images and shared durable storage so RayJobs can run independently of the API process.
12. Make accurate status, bounded retries, cancellation, and cleanup core requirements in each path, with PostgreSQL-to-RayJob reconciliation limited to UI/API runs.