Creation pipeline

It is required a zip archive with the lambda; see lambda creation description and requirements for details.

The first stage of lambda creation is Docker image creation.

Granted zip archive will be supplemented by several files and transformed into another archive which will be stored in s3 storage. Using the Luna-Builder service the archive from s3 will be used by kaniko tool which allows building images in kubernetes cluster independently on kubernetes container runtime.

It is possible to get lambda image creation status during image creation.

Below is a table with possible image creation statuses:

status	description
in_progress	image creation in progress
error	an error occurred during image creation
completed	image creation completed but luna-builder not cleaned up yet
not_found	image creation complete and luna-builder cleaned up or it never was
pending	image creation waiting for suitable conditions

If any error occurred during image creation it is possible to get lambda image creation logs to find and resolve the problem. It is highly recommended to delete lambda before making the next attempt of lambda creation. During lambda image creation, lambda status is waiting.

Completely created image will be pushed to the registry (LAMBDA_REGISTRY specified in the service configuration).

There are several requirements to make this actions possible, see environment requirements for details.

During docker image creation there will be one pod named pod/kaniko-lambda-<lambda_id> in kubernetes cluster, where lambda id matches the lambda id available by user from luna-lambda service.

If lambda created from imported image, creation stage will be skipped

The second stage of lambda creation is to create a service in the kubernetes cluster.

It is possible to set replicas count for lambda in pod_count parameter in deploy_parameters. See create lambda for more details.

Each lambda pod is one replica, see Kubernetes documentation for more information about deployment replicas.

This stage will be executed automatically after image creation is completed.

During lambda creation, it is not possible to get lambda image creation status or get lambda image creation logs.

During lambda creation, it is possible to get lambda status.

Below is a table with possible lambda statuses:

status	description
running	lambda running
waiting	waiting for lambda to start
terminated	lambda stopped
not_found	lambda not found in kubernetes cluster
pending	lambda waiting for suitable conditions

The running status is returned when at least one pod in lambda’s replicas pods is running or when there is only one lambda pod and it’s running.

The waiting status is also returned when all pods in the lambda’s replicas have been created but are still starting up.

The response will include pod_statuses object if at least one of lambda’s pods is in running status.

If any error during lambda creation/initialization, it is possible to get lambda logs to find and resolve the problem.

If lambda status is pending it means that there is no suitable conditions for such lambda, for example lambda requires GPU and kubernetes cluster cannot provides GPU for lambda.

If lambda status is terminated and no logs can be found, it is most likely that there are several problems with registry access from the kubernetes cluster and/or some permission issues.

Completely created service will have running status.

After lambda creation successfully completed kubernetes cluster will contain replicaset, deployment, service and pod named lambda-<lambda-id> where lambda id matches the lambda id available by user from luna-lambda service.

Imported image

Lambda can be created from existing image by providing imported_image parameters in create lambda.

Using imported image can be useful when targeted machine does not have enough computing resources for image build or when it is required to deploy lambda on several product instances.

Base image

Every lambda (except the one created from an imported image) is built on top of a base docker image. There are three built-in base images:

name	usage
lpa-lambda-base	default base image for lambdas and workflows
lpa-lambda-base-fsdk	base image with FSDK, must be passed explicitly in the base_image parameter
lpa-lambda-agent-base	base image for agent-type lambdas

The base image can be set per lambda with the base_image parameter (image name and tag) of the create lambda request. The value is stored with the lambda and is used for all the further image rebuilds of this lambda.

If base_image is not provided, the base image is resolved from the LAMBDA_BASE_IMAGE service setting:

if the request does not contain base_image, the setting entry for the lambda type is used (agent_base_image for agent-type lambdas, base_image for all the others). When the entry is empty, the built-in image for the lambda type with the current version tag is used;

if the request contains base_image with name only, the tag is taken from the setting entry matching the name. When the matching entry is empty, the current version tag is used;

if the request contains base_image with both name and tag, the image is used as is.

For the agent-type lambdas and workflows, a base_image provided without a name (for example, with a tag only) is automatically assigned the lpa-lambda-agent-base name. Agent jobs keep the historical behavior: when no base_image is provided at all, the lpa-lambda-agent-base name is injected, so agent jobs are always built on the agent base image.

The LAMBDA_BASE_IMAGE setting value consists of three entries (base_image, fsdk_base_image, agent_base_image), each of which is an object with name and tag fields or empty (null). The setting is empty by default, in which case all the base images are resolved as described above without the setting.

The setting allows pinning the base images for the platform: when the new base image breaks backward compatibility, the old base image can be fixed in the setting, and the lambdas created (or rebuilt) without an explicit base_image will keep working on the pinned image.

Warning

Lambdas with frozen dependencies (a committed poetry.lock) can break when the base image changes: the locked dependencies were resolved against the system packages of the old base image, and a new base image can ship different versions of the system dependencies (or not include them at all). If the new base image breaks such lambdas, take the old base image (name and tag) and either pin it in the LAMBDA_BASE_IMAGE setting or provide it explicitly in the base_image parameter of every create lambda request.

On service startup all the non-empty entries of the LAMBDA_BASE_IMAGE setting are checked for presence in the LAMBDA_REGISTRY registry (the built-in base image with the current version tag is checked when the setting is empty).

Lambda export and import

Lambda can be exported as zip archive containing lambda docker image, source archive and json with parameters provided with lambda creation request. See export lambda for more info.

Lambda can be imported via import lambda request. Archive provided with request must follow the same structure as result archive from export lambda. Parameters that will be provided with request will overwrite parameters within archive.