Job queue architecture
----------------------

The service manages job lifecycle through two background processes:

-  One periodically retrieves ``current_data`` from running jobs. This is service data
   that allows more flexible job management — the job can use it to track progress,
   store intermediate results, and coordinate with the service updating those data in
   `job context <./workflow_development.html#job-context>`_.

-  The other manages the queue: starts pending jobs and checks completion status.

Regardless of how many service instances are running, only one instance controls
queue management.

Job queue view is avaiable via `get job queue request <./_static/api.html#tag/job-queue>`_.

Jobs are processed in **FIFO** (first-in, first-out) order. Each Kubernetes namespace
has its own job pool with an independent queue. The number of simultaneously running
jobs per namespace is limited by ``parralel_job_limit`` (configured for the default namespace
and for each namespace in ``allowed_namespaces``):

.. code-block::

    # settings for default namespace
    [LAMBDA_RESOURCE_LIMITS.DEFAULT_NAMESPACE]
    # parralel job limit for default namespace
    parralel_job_limit = 20
    # other limits for namespace
    ...
    # settings for other allowed namespaces
    [LAMBDA_RESOURCE_LIMITS.allowed_namespaces.0]
    # example of allowed namespace name
    namespace = allowed-namespace
    # parralel job limit for non-default namespace
    parralel_job_limit = 20
    # other limits for namespace
    ...

Job statuses
------------

Job status — main lifecycle states:

.. list-table::
   :header-rows: 1

   * - Status
     - Description
   * - ``pending``
     - Job is waiting to be started from the queue
   * - ``running``
     - Job is running in Kubernetes
   * - ``done``
     - Job has completed successfully
   * - ``error``
     - Job has finished with an error

Job queue status — last execution status in the queue:

.. list-table::
   :header-rows: 1

   * - Status
     - Description
   * - ``pending``
     - Job has never been executed yet
   * - ``running_partial``
     - Job has partially completed and will be started again (remaining work)
   * - ``completed``
     - Job has fully completed all work
   * - ``insufficient_resources``
     - | Job cannot start due to insufficient resources in the namespace
       | (CPU, RAM, GPU, labels)
   * - ``failed``
     - An error occurred during job execution

Job lifecycle in the queue
--------------------------

1.  A job is created with status ``pending`` and enters the queue.

2.  The service finds it in the queue and attempts to start:

    -  If the pod is not yet running — starts it.
    -  Checks namespace resources (CPU, RAM, GPU, labels) — if insufficient,
       requeues with ``insufficient_resources`` (see `LAMBDA_RESOURCE_LIMITS` for details).

3.  After starting, the status changes to ``running``:

    -  The ``completeness`` endpoint is checked periodically — if the job reports
       that it has finished all work, the job is done → status ``done``.
    -  The ``max_operating_time`` is checked — if exceeded, the job is stopped
       and requeued with ``running_partial``.
    -  The ``execution_completeness`` and ``busy`` endpoints are checked — if
       the job reports partial completion and is not busy, it is requeued
       with ``running_partial``.

4.  On errors — requeued with ``failed``.

Queue management via HTTP API
-----------------------------

When creating or updating a job/workflow via the API, the ``deploy_parameters``
includes the ``allow_insufficient_resources`` parameter (default ``0`` / ``false``):

-  **``allow_insufficient_resources = 0``** (default) — if the namespace does not have
   sufficient resources (CPU, RAM, GPU, labels), the job **will not be created** or will
   be rejected. The service returns an error.

-  **``allow_insufficient_resources = 1``** — the job will be created and added to the
   queue even if resources are insufficient. The job will wait in the queue
   (``insufficient_resources``) until resources become available, then start.

Configuration parameters
------------------------

.. list-table::
   :header-rows: 1

   * - Parameter
     - Description
   * - ``LUNA_LAMBDA_JOBS.jobs_refresh_period``
     - Interval for updating ``current_data`` of running jobs in seconds
   * - ``LUNA_LAMBDA_JOBS.max_operating_time``
     - | Maximum job operating time in seconds. After expiration, the job
       | is stopped and requeued with ``running_partial``.
   * - ``LUNA_LAMBDA_INACTIVE_JOB_CLEAR_INTERVAL``
     - | Periodic check and disable of inactive jobs.
       | Contains: ``interval``, ``interval_type``,
       | ``check_interval``, ``active``.