Creation pipeline ================= #) It is required a zip archive with the lambda; see `lambda creation description and requirements `_ for details. #) The first stage of lambda creation is Docker image creation. Granted zip archive will be supplemented by several files and transformed into another archive which will be stored in `s3 storage <./lambda_requirements.html#s3-storage-requirements>`_. Using the Luna-Builder service the archive from s3 will be used by `kaniko tool `_ which allows building images in kubernetes cluster independently on `kubernetes container runtime `_. It is possible to `get lambda image creation status <./_static/api.html#operation/getLambdaImageStatus>`_ during image creation. Below is a table with possible image creation statuses: +-------------+---------------------------------------------------------------------+ | status | description | +=============+=====================================================================+ | in_progress | image creation in progress | +-------------+---------------------------------------------------------------------+ | error | an error occurred during image creation | +-------------+---------------------------------------------------------------------+ | completed | image creation completed but luna-builder not cleaned up yet | +-------------+---------------------------------------------------------------------+ | not_found | image creation complete and luna-builder cleaned up or it never was | +-------------+---------------------------------------------------------------------+ | pending | image creation waiting for suitable conditions | +-------------+---------------------------------------------------------------------+ If any error occurred during image creation it is possible to `get lambda image creation logs <./_static/api.html#operation/getLambdaImageCreationLogs>`_ to find and resolve the problem. It is highly recommended to `delete lambda <./_static/api.html#operation/deleteLambda>`_ before making the next attempt of lambda creation. During lambda image creation, `lambda status <./_static/api.html#operation/getLambdaStatus>`_ is `waiting`. Completely created image will be pushed to the registry (`LAMBDA_REGISTRY` specified in the `service configuration <./config.html#configuration-requirements>`_). There are several requirements to make this actions possible, see `environment requirements <./lambda_requirements.html>`_ for details. During docker image creation there will be one pod named `pod/kaniko-lambda-` in kubernetes cluster, where lambda id matches the lambda id available by user from `luna-lambda` service. **If lambda created from imported image, creation stage will be skipped** #) The second stage of lambda creation is to create a service in the kubernetes cluster. It is possible to set replicas count for lambda in `pod_count` parameter in `deploy_parameters`. See `create lambda <./_static/api.html#operation/createLambda>`_ for more details. Each lambda pod is one replica, see `Kubernetes documentation `_ for more information about deployment replicas. This stage will be executed automatically after image creation is completed. During lambda creation, it is not possible to `get lambda image creation status <./_static/api.html#operation/getLambdaImageStatus>`_ or `get lambda image creation logs <./_static/api.html#operation/getLambdaImageCreationLogs>`_. During lambda creation, it is possible to `get lambda status <./_static/api.html#operation/getLambdaStatus>`_. Below is a table with possible lambda statuses: +------------+----------------------------------------+ | status | description | +============+========================================+ | running | lambda running | +------------+----------------------------------------+ | waiting | waiting for lambda to start | +------------+----------------------------------------+ | terminated | lambda stopped | +------------+----------------------------------------+ | not_found | lambda not found in kubernetes cluster | +------------+----------------------------------------+ | pending | lambda waiting for suitable conditions | +------------+----------------------------------------+ The `running` status is returned when at least one pod in lambda's replicas pods is running or when there is only one lambda pod and it's running. The `waiting` status is also returned when all pods in the lambda's replicas have been created but are still starting up. The response will include `pod_statuses` object if at least one of lambda's pods is in `running` status. If any error during lambda creation/initialization, it is possible to `get lambda logs <./_static/api.html#operation/getLambdaLogs>`_ to find and resolve the problem. If lambda status is pending it means that there is no suitable conditions for such lambda, for example lambda requires GPU and kubernetes cluster cannot provides GPU for lambda. If lambda status is terminated and no logs can be found, it is most likely that there are several problems with registry access from the kubernetes cluster and/or some permission issues. Completely created service will have `running` status. After lambda creation successfully completed kubernetes cluster will contain replicaset, deployment, service and pod named `lambda-` where lambda id matches the lambda id available by user from `luna-lambda` service. Imported image -------------- Lambda can be created from existing image by providing `imported_image` parameters in `create lambda <./_static/api.html#operation/createLambda>`_. Using imported image can be useful when targeted machine does not have enough computing resources for image build or when it is required to deploy lambda on several product instances. Base image ---------- Every lambda (except the one created from an imported image) is built on top of a base docker image. There are three built-in base images: +-----------------------+-----------------------------------------------+ | name | usage | +=======================+===============================================+ | lpa-lambda-base | default base image for lambdas and workflows | +-----------------------+-----------------------------------------------+ | lpa-lambda-base-fsdk | base image with FSDK, must be passed | | | explicitly in the `base_image` parameter | +-----------------------+-----------------------------------------------+ | lpa-lambda-agent-base | base image for agent-type lambdas | +-----------------------+-----------------------------------------------+ The base image can be set per lambda with the `base_image` parameter (image `name` and `tag`) of the `create lambda <./_static/api.html#operation/createLambda>`_ request. The value is stored with the lambda and is used for all the further image rebuilds of this lambda. If `base_image` is not provided, the base image is resolved from the `LAMBDA_BASE_IMAGE` `service setting <./config.html#configuration-requirements>`_: * if the request does not contain `base_image`, the setting entry for the lambda type is used (`agent_base_image` for agent-type lambdas, `base_image` for all the others). When the entry is empty, the built-in image for the lambda type with the current version tag is used; * if the request contains `base_image` with `name` only, the tag is taken from the setting entry matching the name. When the matching entry is empty, the current version tag is used; * if the request contains `base_image` with both `name` and `tag`, the image is used as is. For the *agent*-type lambdas and workflows, a ``base_image`` provided without a ``name`` (for example, with a tag only) is automatically assigned the ``lpa-lambda-agent-base`` name. Agent jobs keep the historical behavior: when no ``base_image`` is provided at all, the ``lpa-lambda-agent-base`` name is injected, so agent jobs are always built on the agent base image. The `LAMBDA_BASE_IMAGE` setting value consists of three entries (`base_image`, `fsdk_base_image`, `agent_base_image`), each of which is an object with `name` and `tag` fields or empty (null). The setting is empty by default, in which case all the base images are resolved as described above without the setting. The setting allows pinning the base images for the platform: when the new base image breaks backward compatibility, the old base image can be fixed in the setting, and the lambdas created (or rebuilt) without an explicit `base_image` will keep working on the pinned image. .. warning:: Lambdas with frozen dependencies (a committed ``poetry.lock``) can break when the base image changes: the locked dependencies were resolved against the system packages of the old base image, and a new base image can ship different versions of the system dependencies (or not include them at all). If the new base image breaks such lambdas, take the old base image (name and tag) and either pin it in the `LAMBDA_BASE_IMAGE` setting or provide it explicitly in the ``base_image`` parameter of every `create lambda <./_static/api.html#operation/createLambda>`_ request. On service startup all the non-empty entries of the `LAMBDA_BASE_IMAGE` setting are checked for presence in the `LAMBDA_REGISTRY` registry (the built-in base image with the current version tag is checked when the setting is empty). Lambda export and import ------------------------ Lambda can be exported as zip archive containing lambda docker image, source archive and json with parameters provided with lambda creation request. See `export lambda <./_static/api.html#operation/exportLambda>`_ for more info. Lambda can be imported via `import lambda <./_static/api.html#operation/exportLambda>`_ request. Archive provided with request must follow the same structure as result archive from `export lambda <./_static/api.html#operation/exportLambda>`_. Parameters that will be provided with request will overwrite parameters within archive.