# Proposed Ray Architecture for CheckMAITE

# Summary

CheckMAITE has two related but different workload types:

1. Models that should remain available for inference, regardless of whether anyone has a notebook open.
2. Interactive capability executions submitted from notebooks, often using datasets, models, or Python objects already created in the notebook.

Running both workload types on one shared Ray cluster is convenient, but it creates several problems. A large CheckMAITE execution can consume resources needed for inference. A model-serving upgrade can replace the Ray cluster while CheckMAITE work is still running. Everyone with direct access to the cluster is effectively part of the same trust group. It is also difficult to determine which Ray resources belong to which CheckMAITE execution.

The proposed design separates these responsibilities:

• A dedicated model-serving environment keeps shared models available and protects their capacity from notebook work.
• Each trusted user group receives its own persistent interactive compute environment for CheckMAITE.
• Users continue submitting normal Python objects directly from notebooks, without first converting every execution into a batch script.
• A small CheckMAITE job directory records who owns each execution, where it is running, its status, and where its results are stored. This allows users and operators to find and manage work after a notebook closes or crashes.

This preserves the interactive workflow that makes CheckMAITE convenient while reducing interference between model serving, user groups, and capability executions.

# 1. Requirements

# User requirements

• Users can submit CheckMAITE capabilities directly from Python notebooks.
• Users can pass serializable capability, dataset, model, and configuration objects they have already created.
• Submitting work is fast enough to feel interactive.
• Closing or losing a notebook does not automatically stop work that has already been accepted.
• Users can reconnect and list, inspect, cancel, or retrieve their executions.
• Common models remain loaded and ready for inference.
• Capabilities such as NRTK can repeatedly request predictions for generated or perturbed data.
• User groups have separate capacity and cannot access each other’s Ray environments.
• Results, reports, analytics, and provenance are stored outside short-lived worker processes.

# Operational requirements

• Notebook work should not slow down or interrupt shared model inference.
• Each group has clear limits on how much CPU, memory, and GPU capacity it can use.
• Operators can identify which resources belong to a particular CheckMAITE execution.
• Updating model-serving infrastructure does not terminate unrelated CheckMAITE work.
• A notebook crash does not remove the only way to find or cancel a running execution.
• Finished and abandoned Ray processes are cleaned up automatically.
• Users receive a clear distinction between work waiting for resources and work that is actually running.

# 2. Ray and CheckMAITE terminology

The word job is overloaded:

• A CheckMAITE job, called a CheckMAITE execution in this document, is one submitted capability run. It has a CheckMAITE ID, status, result, report, and analytics records.
• A Ray task is the remote Python function that performs some or all of that execution.
• A Ray actor is a long-running process supporting the execution, such as its controller or model.
• A Ray driver job is the notebook or Python process connected to Ray. The Ray Dashboard may show several CheckMAITE executions under one driver ID because they came from the same notebook connection.
• A Ray Jobs API job is a separate Ray feature that starts a command or script on a cluster. It is discussed later but is not the primary mechanism proposed here.

These identifiers should not be treated as interchangeable. The user-facing identifier should always be the CheckMAITE execution ID, with Ray identifiers available as supporting diagnostic information.

# 3. High-level architecture

flowchart TB
    subgraph Users["Interactive users"]
        JA["Jupyter / Python — Group A"]
        JB["Jupyter / Python — Group B"]
    end

    subgraph Directory["CheckMAITE job directory"]
        API["Ownership, status, discovery, and cancellation API"]
        DB[("Execution records")]
        STORE[("Reports, analytics, and results")]
        API --- DB
    end

    subgraph Serving["Shared model-serving environment"]
        GW["Authenticated inference endpoint"]
        RS["RayService"]
        MODELS["Always-on shared models"]
        GW --> RS --> MODELS
    end

    subgraph GroupA["Group A interactive compute environment"]
        RCA["Persistent RayCluster A"]
        REGA["CheckMAITE registry"]
        CTRLA["Execution controllers"]
        TASKA["Capability computations"]
        MODELA["Group/job-specific models"]
        RCA --- REGA
        REGA --> CTRLA --> TASKA
        TASKA --> MODELA
    end

    subgraph GroupB["Group B interactive compute environment"]
        RCB["Persistent RayCluster B"]
        REGB["CheckMAITE registry"]
        CTRLB["Execution controllers"]
        TASKB["Capability computations"]
        MODELB["Group/job-specific models"]
        RCB --- REGB
        REGB --> CTRLB --> TASKB
        TASKB --> MODELB
    end

    JA -->|"Interactive Python submission"| RCA
    JB -->|"Interactive Python submission"| RCB
    JA --> API
    JB --> API
    CTRLA --> API
    CTRLB --> API
    TASKA --> STORE
    TASKB --> STORE
    TASKA -->|"Shared-model inference"| GW
    TASKB -->|"Shared-model inference"| GW

The serving environment and interactive compute environments can run in the same Kubernetes cluster, but they use different Ray clusters, namespaces, credentials, and resource limits.

# 4. Shared model-serving environment

# Purpose

This environment hosts models that should remain available even when no notebook or CheckMAITE execution is active. Examples include an approved reference model used by several evaluations or a model exposed to other applications for online inference.

A dedicated KubeRay RayService is appropriate because it manages model replicas, monitors their health, scales them in response to demand, and supports updates without intentionally interrupting inference traffic.

# Access and authorization

Users do not connect their notebooks directly to the serving Ray cluster. They access only a controlled inference endpoint.

CheckMAITE computation
  → authenticated inference request
  → serving proxy
  → approved model
  → prediction response

The inference endpoint identifies the calling user or group and can enforce:

• which models the group may use;
• request and concurrency limits;
• audit logging;
• model-version selection;
• payload-size limits;
• access to sensitive models.

For example:

Group A
  - can call models reference-model-v1 and detector-v3
  - cannot view the serving dashboard
  - cannot deploy, replace, or delete a model
  - cannot access Group B's restricted model

Group B
  - can call reference-model-v1 and group-b-model-v2
  - cannot connect to Group A's interactive cluster
  - cannot access the serving cluster's Ray Client or internal control ports

Serving administrators
  - can deploy and update approved models
  - can inspect serving health and capacity

Authentication should occur at an API gateway or internal service endpoint. Direct access to the serving cluster’s Ray Client, dashboard API, and internal control services should be blocked from notebook environments.

# Illustrative capacity

Capacity depends more on model cost, latency, batching, and simultaneous requests than on the total number of registered users. Ten users making occasional requests may need less capacity than one NRTK execution sending hundreds of images at once.

An illustrative CPU serving configuration might be:

Head/control pod:
  1–2 Kubernetes CPUs and 4–8 GiB memory
  0 Ray CPUs available for model or user work

Serving workers:
  minimum 2 workers, each 2 CPUs and 4–8 GiB
  enough room for at least 2 model replicas

Model replicas:
  minimum 2 replicas for availability
  1 CPU each for a lightweight CPU model
  autoscale to 4–8 replicas after load testing

For a GPU model, the equivalent baseline might be two replicas on two GPU workers so that one pod or node failure does not remove all serving capacity.

These are examples, not final sizing recommendations. Representative inference tests should determine replica CPU/GPU requirements, maximum concurrent requests, and useful batch sizes.

A RayService update can temporarily run both the old and replacement clusters. Kubernetes must therefore have enough spare capacity—or autoscaling headroom—to host roughly twice the normal serving footprint during an upgrade.

# Relationship to the current deployment

The current COSMOS development deployment does not separate serving and CheckMAITE compute:

• one RayService supplies both the Serve application and general Ray Client access;
• the head advertises CPUs for user work;
• the healthz model replica runs on the head;
• CheckMAITE tasks may run on either the head or worker;
• both Ray pods currently run on the same physical Kubernetes node;
• the active cluster has one worker despite the Git configuration requesting four;
• serving and CheckMAITE work therefore compete for the same four logical Ray CPUs.

The proposed design replaces that shared pool with dedicated serving capacity and separate interactive capacity. Kubernetes node selectors, taints/tolerations, or Ray custom resources should ensure serving replicas use serving workers and CheckMAITE computations use interactive compute workers.

# Model versioning

Each CheckMAITE execution should record the exact model revision it used. A model update must not cause one execution to receive predictions from two different revisions.

Versioned endpoints could look conceptually like:

/models/detector/v3/predict
/models/detector/v4/predict

An execution remains pinned to one version even if a newer model becomes available during the run.

# 5. Interactive CheckMAITE environments

# One environment per trusted user group

Each trusted group receives a persistent KubeRay RayCluster in its own Kubernetes namespace.

Group A → checkmaite-group-a namespace → RayCluster A
Group B → checkmaite-group-b namespace → RayCluster B

Members of one group are mutually trusted at the Ray level. If users within a group must not be able to interfere with each other, the isolation unit needs to become one RayCluster per user rather than per group.

# Why the clusters remain running

A persistent group cluster preserves the notebook experience:

• the notebook can send Python objects directly to Ray;
• users do not wait for a new cluster on every submission;
• existing CheckMAITE executions remain discoverable after a notebook reconnects;
• loaded job-specific models can be reused where appropriate;
• each group can have its own package versions and resource limits;
• worker capacity can still grow and shrink within configured limits.

# Illustrative layout and sizing

A small interactive group environment might start with:

Head pod:
  1–2 Kubernetes CPUs and 4 GiB memory
  0 Ray CPUs for user computations

Control capacity:
  either reserved resources on the head
  or 1 small static control worker with 1 CPU and 2–4 GiB

Compute workers:
  minimum 1 warm worker with 2 CPUs and 4–8 GiB
  autoscale to 3–6 workers according to the group's quota

Optional GPU workers:
  minimum 0
  autoscale up to the number of GPUs assigned to the group

The head still needs Kubernetes CPU even when it advertises zero Ray CPUs. Zero Ray CPUs means that Ray will not place ordinary model or capability work there; it does not mean the head process uses no CPU.

The CheckMAITE control actors are much smaller than initially assumed in the Slack discussion:

Registry actor:
  explicitly 0 Ray CPUs in the proposed configuration

Controller actor:
  approximately 0.01 Ray CPU per active execution

Typical capability worker:
  1 Ray CPU unless overridden

A normal execution therefore needs approximately 1.01 logical CPUs while its worker is active, not three CPUs. Two concurrent one-CPU workers need a little more than two logical CPUs because their controllers also reserve a small amount.

The exact worker count should be based on desired concurrency. For example, if a group must run four one-CPU capabilities simultaneously, it needs at least four compute CPUs plus control-plane and failure headroom. It should not be sized to exactly 4.04 CPUs with no spare capacity.

# Keeping control actors off autoscaled workers

Ray will not automatically understand that a registry or controller is control-plane infrastructure. Placement must be explicit.

Two practical options are:

1. Reserve a custom Ray resource on the head, such as checkmaite-control, and require registry/controller actors to request it while requesting little or no normal CPU.
2. Run a small, fixed control worker group that advertises checkmaite-control, while ordinary autoscaled workers do not.

This matters because Ray does not remove a worker while an actor is still running on it. If a detached registry lands on an ordinary autoscaled worker, that worker may remain running forever even when no capability computations are active.

At least one compute worker should remain warm when low interactive latency is important. A scale-to-zero compute group saves cost but adds pod scheduling, image startup, and package-import delays to the next submission.

# 6. CheckMAITE execution lifecycle

sequenceDiagram
    participant N as Notebook
    participant R as Group registry
    participant C as Execution controller
    participant W as Capability computation
    participant M as Model
    participant D as Job directory/result store

    N->>R: Submit or find matching execution
    R-->>N: Return CheckMAITE execution ID
    N->>C: Transfer capability, dataset, model, and configuration
    C->>R: Record that the execution is being scheduled
    C->>W: Start capability computation
    W->>M: Request predictions as needed
    M-->>W: Return predictions
    W->>D: Store analytics and results
    W-->>C: Return result reference
    C->>R: Record final state
    C->>D: Publish status and result location
    R-->>N: Support status, list, cancel, and result requests

# Required behavior

1. Each group cluster has one default CheckMAITE registry rather than one registry per notebook run.
2. Stable user/workspace scopes organize and deduplicate records inside that registry.
3. The execution controller takes responsibility for the worker before submission reports success.
4. Users can distinguish SCHEDULING from RUNNING.
5. Waiting-for-resources timeouts are separate from execution timeouts.
6. A result is reported as complete only after the shared status record has been updated.
7. A background cleanup process removes expired terminal controllers even if no one submits another job.
8. Ray actor and task names include the CheckMAITE execution ID for troubleshooting.
9. A small external job directory records ownership, status, cluster location, and result location so operators are not dependent on one notebook or Ray dashboard.

# Notebook disconnection

After the controller has accepted responsibility for the work, losing the notebook does not stop the execution. A replacement notebook reconnects to the same group cluster and stable scope:

configure_job_backend(
    "ray",
    address=group_cluster_address,
    idempotency_scope=stable_workspace_scope,
)

jobs = list_jobs()

If the entire group RayCluster is lost, arbitrary in-memory notebook objects cannot always be recreated. The job directory should mark those executions as lost or failed. Executions whose inputs also have durable references may optionally be resubmitted.

# 7. Model execution patterns

# Shared approved model

Use the dedicated serving environment:

NRTK computation
  → generate perturbed image batch
  → send inference request
  → shared versioned model
  → receive predictions

NRTK can generate many model requests, so requests should be batched and concurrency should be bounded. Sending every image as an individual base64-encoded JSON request is likely to create unnecessary network and serialization overhead.

# Model supplied from a notebook

A model created or loaded in a notebook can be sent through the interactive Ray connection and loaded into a model actor in that group’s cluster.

Notebook model object
  → group RayCluster
  → long-running model process
  → CheckMAITE computation requests predictions

This avoids crossing from the group cluster to the shared serving cluster and preserves the convenience of passing live Python objects.

A full Ray Serve deployment is unnecessary when only one CheckMAITE execution needs the model. A model actor is simply a stateful Python process that keeps the model loaded for repeated calls.

# Model reused by a group

A model used by several executions in one trusted group can remain in a named group-owned actor. It should have:

• an immutable model/version identity;
• a declared CPU or GPU requirement;
• ownership information;
• a lease or reference count;
• an idle timeout;
• explicit cleanup behavior.

# Group-specific model service

If a group needs its own HTTP endpoint, independently scalable replicas, or request batching shared across many executions, deploy a separate RayService for that group. Arbitrary user models should not be deployed into the globally shared serving environment.

# 8. Isolation and authorization

Ray does not provide strong user isolation inside one Ray cluster. A user who can connect through Ray Client can execute arbitrary Python in that cluster. Separate Ray clusters are therefore the meaningful Ray-level isolation boundary.

# Is a Kubernetes namespace a security boundary?

A namespace is not a security boundary by itself. By default, workloads in different namespaces may still communicate, and permissions may still be granted too broadly.

A namespace becomes part of an effective isolation boundary only when it is combined with:

• Kubernetes RBAC limiting which users and service accounts can access resources in it;
• NetworkPolicy blocking traffic from other namespaces;
• a dedicated service account with minimal permissions;
• separate Ray credentials;
• ResourceQuota and LimitRange;
• controlled access to storage credentials and object-store prefixes;
• authenticated dashboard and management routes.

The isolation boundary in this proposal is therefore:

separate RayCluster
+ separate namespace
+ RBAC
+ NetworkPolicy
+ separate credentials
+ resource and storage limits

A Ray namespace is different from a Kubernetes namespace. A Ray namespace only organizes names and actors inside one Ray cluster; it does not enforce network access, permissions, or resource quotas.

# Serving isolation

The serving environment exposes only approved inference endpoints to user groups. Notebook users cannot directly access its Ray Client, internal cluster control service, or deployment administration APIs.

Each inference request carries a user or group identity. The gateway can then enforce model permissions, rate limits, audit records, and group-specific routes.

# Execution ownership

The CheckMAITE job directory stores the owner and group for each execution. It controls what users see through CheckMAITE’s list, inspect, and cancel interfaces.

This improves normal user and operator behavior, but members of one group remain mutually trusted because they have direct access to the same Ray cluster. Preventing one member from using low-level Ray APIs against another member’s resources requires a separate cluster per user.

# 9. Scaling and scheduling

# Shared serving environment

Scaling occurs in layers:

More inference requests
  → start more model replicas
  → start more Ray worker pods if needed
  → start more EKS nodes if the pods do not fit

An illustrative starting point for a lightweight CPU model is:

2 model replicas minimum
1 CPU per replica
2 serving workers minimum
4–8 replicas maximum after load testing

A service that must survive one worker failure should not place all minimum replicas on one pod or physical node. GPU models need the same consideration at the GPU-node level.

Capacity should be measured in concurrent inference requests and model throughput, not simply number of users. Dynamic batching may allow one replica to serve many users efficiently.

# Interactive group environment

Scaling is driven by CheckMAITE computations and group-specific models:

More capability/model work
  → Ray requests more compute workers
  → Kubernetes schedules more worker pods
  → EKS adds CPU or GPU nodes if required

A small group might use:

1 warm 2-CPU worker minimum
3 workers / 6 CPUs maximum
0 GPU workers minimum
1 GPU worker maximum

A larger group or a group running several NRTK evaluations would require a higher maximum. The key settings are:

• desired number of simultaneous CheckMAITE executions;
• CPU/GPU requirement per execution;
• memory required by datasets and models;
• acceptable queue time;
• acceptable cost when idle.

Kubernetes quotas limit the total size of each group cluster. Ray decides how tasks use that capacity internally. Because Kubernetes sees Ray worker pods rather than each individual CheckMAITE execution, CheckMAITE itself must provide per-user concurrency limits, pending limits, and fair-use policies inside the group.

# Admission and backpressure

CheckMAITE should enforce:

• maximum active executions per group and user;
• maximum queued or scheduling executions;
• validation of CPU, memory, and GPU requests;
• a timeout for work that cannot obtain resources;
• limits on simultaneous requests to shared models;
• clear explanations when a request cannot fit on any configured worker type.

# 10. Failure and upgrade behavior

Keeping serving and interactive computation separate is important during upgrades. RayService decides that an old cluster can be removed when the replacement model service is healthy. It does not wait for unrelated CheckMAITE computations that happen to be running on the old cluster.

# 11. Visibility and operations

Users and operators should not need to understand several unrelated Ray IDs to answer basic questions such as:

• Is my CheckMAITE execution waiting or running?
• Which model version is it using?
• Why has it not started?
• Can I cancel it after my notebook crashed?
• Is it using shared serving capacity or a job-specific model?
• Which group owns it?

CheckMAITE should present one execution ID and one coherent status. Ray actor, task, driver, pod, and node identifiers should remain available behind that execution for troubleshooting rather than becoming part of the normal user workflow.

The management view should show:

• owner and group;
• current state and how long it has been in that state;
• requested resources and whether they are available;
• model name and immutable version;
• result and report location;
• clear completion, cancellation, timeout, or failure reason.

Operators also need summary views showing group capacity, queued work, stale controllers, serving saturation, and resources that remain after executions finish. Alerts should focus on user-impacting conditions rather than raw actor counts alone.

# 12. Ray Jobs API as an alternative

Ray also provides a feature called the Ray Jobs API. It allows a client to submit a command or script to a Ray cluster, then inspect its status, stream logs, or stop it later. Unlike Ray Client, the submitted work is naturally independent of the notebook connection.

A Ray Jobs submission looks conceptually like:

python -m checkmaite.runner --submission-id abc123

This could support another CheckMAITE backend in the future. It would provide standard Ray job identifiers, logs, and cancellation and could be useful for reproducible, noninteractive, scheduled, or long-running executions.

It is not the primary choice for this proposal because it does not directly accept the capability, dataset, model, and other live Python objects already present in notebook memory. Users would first need to package code and stage inputs somewhere the script could load them. That would change the current interactive workflow and add startup and preparation time.

The proposed architecture therefore uses Ray Client for the main notebook experience and keeps Ray Jobs as an optional future backend for use cases where script-based, independently packaged execution is preferable.

# 13. Tradeoffs

# Benefits

• Preserves the current interactive Python experience.
• Keeps shared models available independently of notebook activity.
• Prevents routine serving upgrades from terminating CheckMAITE executions.
• Prevents notebook computations from consuming reserved serving capacity.
• Provides meaningful isolation and independent quotas between trusted groups.
• Supports both shared approved models and arbitrary notebook-provided models.
• Allows serving and interactive compute to scale independently.
• Allows users to reconnect to executions after losing a notebook.

# Costs and limitations

• CheckMAITE must continue maintaining and improving its registry/controller lifecycle.
• Every group cluster has a small always-running head/control cost.
• Members of one group remain mutually trusted within their Ray cluster.
• Passing arbitrary Python objects requires compatible Python, Ray, and CheckMAITE environments.
• Objects containing notebook-local file paths still require equivalent worker mounts or durable storage.
• Calls from a group cluster to a shared model require network serialization and careful batching.
• Losing an entire Ray cluster cannot reconstruct inputs that existed only in notebook memory.
• Kubernetes quotas limit the group cluster as a whole, but they cannot directly prioritize or fairly queue individual CheckMAITE executions inside Ray. CheckMAITE must implement those user-level limits and policies.

# 14. Key decisions

1. Use a dedicated RayService for managed, always-on model serving.
2. Use persistent RayClusters for interactive CheckMAITE execution.
3. Use one RayCluster per trusted user group as the primary isolation boundary.
4. Combine namespaces with RBAC, NetworkPolicy, separate credentials, and quotas; a namespace alone is not isolation.
5. Retain Ray Client so notebooks can submit live Python objects with low latency.
6. Retain and harden CheckMAITE’s registry and controller execution model.
7. Use job-specific model actors for models supplied directly from notebooks.
8. Use authenticated inference endpoints for shared approved models.
9. Keep execution ownership, status, and result locations in a CheckMAITE job directory outside notebook memory.
10. Treat Ray Jobs as an optional future backend rather than the primary interactive path.