Job queue architecture

The service manages job lifecycle through two background processes:

One periodically retrieves current_data from running jobs. This is service data that allows more flexible job management — the job can use it to track progress, store intermediate results, and coordinate with the service updating those data in job context.
The other manages the queue: starts pending jobs and checks completion status.

Regardless of how many service instances are running, only one instance controls queue management.

Job queue view is avaiable via get job queue request.

Jobs are processed in FIFO (first-in, first-out) order. Each Kubernetes namespace has its own job pool with an independent queue. The number of simultaneously running jobs per namespace is limited by parralel_job_limit (configured for the default namespace and for each namespace in allowed_namespaces):

# settings for default namespace
[LAMBDA_RESOURCE_LIMITS.DEFAULT_NAMESPACE]
# parralel job limit for default namespace
parralel_job_limit = 20
# other limits for namespace
...
# settings for other allowed namespaces
[LAMBDA_RESOURCE_LIMITS.allowed_namespaces.0]
# example of allowed namespace name
namespace = allowed-namespace
# parralel job limit for non-default namespace
parralel_job_limit = 20
# other limits for namespace
...

Job statuses

Job status — main lifecycle states:

Status	Description
`pending`	Job is waiting to be started from the queue
`running`	Job is running in Kubernetes
`done`	Job has completed successfully
`error`	Job has finished with an error

Job queue status — last execution status in the queue:

Status	Description
`pending`	Job has never been executed yet
`running_partial`	Job has partially completed and will be started again (remaining work)
`completed`	Job has fully completed all work
`insufficient_resources`	Job cannot start due to insufficient resources in the namespace (CPU, RAM, GPU, labels)
`failed`	An error occurred during job execution

Job lifecycle in the queue

A job is created with status pending and enters the queue.
The service finds it in the queue and attempts to start:
- If the pod is not yet running — starts it.
- Checks namespace resources (CPU, RAM, GPU, labels) — if insufficient, requeues with insufficient_resources (see LAMBDA_RESOURCE_LIMITS for details).
After starting, the status changes to running:
- The completeness endpoint is checked periodically — if the job reports that it has finished all work, the job is done → status done.
- The max_operating_time is checked — if exceeded, the job is stopped and requeued with running_partial.
- The execution_completeness and busy endpoints are checked — if the job reports partial completion and is not busy, it is requeued with running_partial.
On errors — requeued with failed.

Queue management via HTTP API

When creating or updating a job/workflow via the API, the deploy_parameters includes the allow_insufficient_resources parameter (default 0 / false):

``allow_insufficient_resources = 0`` (default) — if the namespace does not have sufficient resources (CPU, RAM, GPU, labels), the job will not be created or will be rejected. The service returns an error.
``allow_insufficient_resources = 1`` — the job will be created and added to the queue even if resources are insufficient. The job will wait in the queue (insufficient_resources) until resources become available, then start.

Configuration parameters

Parameter	Description
`LUNA_LAMBDA_JOBS.jobs_refresh_period`	Interval for updating `current_data` of running jobs in seconds
`LUNA_LAMBDA_JOBS.max_operating_time`	Maximum job operating time in seconds. After expiration, the job is stopped and requeued with `running_partial`.
`LUNA_LAMBDA_INACTIVE_JOB_CLEAR_INTERVAL`	Periodic check and disable of inactive jobs. Contains: `interval`, `interval_type`, `check_interval`, `active`.