Monitoring

Data for monitoring

Now we monitor two types of events for monitoring: request and error. First type is all requests, second is failed requests only.

Comparison of data formats for Clickhouse and InfluxDB:

InfluxDB: Each event is presented as a “point” in a time series. The structure of a point includes:

• series name

• start event time

• tags, indexed data in storage, dictionary: keys - string tag names, values - string, integer, float

• fields, non indexed data in storage, dictionary: keys - string tag names, values - string, integer, float

Clickhouse: In Clickhouse, the data structure resembles that of a traditional SQL table. Each event is represented as a record, where:

• The `time` field contains the record’s creation timestamp;

• The `data` field contains a JSON object with all the information that would otherwise be distributed across tags and fields in InfluxDB.

Important: In Clickhouse, there is no differentiation between “tags” and “fields”—all data is consolidated into a single JSON object within the data field.

Monitoring series

The structure and the meaning of each monitoring series remain consistent. However, for Clickhouse, data from tags and fields are merged into a single JSON object under the data field. Below are examples for each series:

• Requests series.

  Triggered on every request. Each point contains a data about corresponding request (execution time and etc).

  InfluxDB:

  tag name	description
  service	always “luna-admin”
  route	concatenation of a request method and a request resource (GET:/accounts)
  status_code	http status code of response

  • fields

    fields	description
    request_id	request id
    execution_time	request execution time

  ClickHouse JSON `data` field Example:

  {
      "service": "luna-admin",
      "route": "POST:/accounts",
      "status_code": 200,
      "request_id": "1536751345,6a5c2191-3e9b-f5a4-fc45-3abf43625c5f",
      "execution_time": 123.45
  }
  

• Errors series.

  Triggered on failed request. Each point contains error_code of luna error.

  InfluxDB:

  • tags

    tag name	description
    service	always “luna-admin”
    route	concatenation of a request method and a request resource (GET:/accounts)
    status_code	http status code of response
    error_code	luna error code

  • fields

    fields	description
    request_id	request id

  ClickHouse JSON `data` field Example:

  {
      "service": "luna-admin",
      "route": "POST:/accounts",
      "status_code": 400,
      "error_code": 13037,
      "request_id": "1536751345,6a5c2191-3e9b-f5a4-fc45-3abf43625c5f",
  }
  

Every handler can add additional tags or fields.

Database

You can refer to documentation for influx database and clickhouse database to compare the databases and choose what benefit your needs more. Note that clickhouse might be the better choice for aggregation You can setup your database credentials in configuration file in section “monitoring”.

Aggregated statistics

For statistical purposes monitoring data is being aggregated and downsampled. Data includes average response time, total responses count, 95 and 99 percentiles, errors counts, sdk estimators usage. To enable aggregation you need to set MONITORING_AGGREGATION setting to 1

InfluxDB:

To perform this task influx tasks are used. Each task runs once per day, fetches all points from the previous day and performs required calculations. To change schedule either edit flux scripts in base_scripts/flux before creating tasks or edit tasks in the Influx UI after creating them. Resulting data is stored in the bucket luna_monitoring_aggregated, its created along with the tasks. To create tasks run use storages tool.

ClickHouse

ClickHouseDB has a built-in mechanism for data aggregation called Materialised views, which allows collecting statistics on the fly.

System info statistics (stats)

GET /4/luna_sys_info (see the getSystemInfo operation) returns, among other things, a stats object built from the aggregated data described above. It’s null when monitoring is disabled, the storage adapter isn’t initialized, or MONITORING_AGGREGATION is off (see app.handlers.sys_info_handler.SysInfoHandler.buildUsageStats). Otherwise it’s assembled month-by-month by app.utils.clickhouse_helper.ClickhouseSysInfoGetter (ClickHouse backend) or app.utils.influx_helper.InfluxSysInfoGetter (InfluxDB backend) — same shape either way, only the source queries differ (app.utils.query_helpers). Zero counts are dropped to keep the payload small.

stats has four top-level lists:

• routes_stats — one entry per (service, route, month), with:

  • request_stats — list of {status_code, count}: how many requests of that status code the route got that month.

  • errors — list of {error_code, count}: how many failed requests of that LUNA error code the route got that month.

  • response_time — list of {status_code, avg, p95, p99}: average, 95th and 99th percentile execution time (in seconds) for that route/status code that month. This merges what used to be three separate series (response_time, response_time_p95, response_time_p99) into one field per status code.

• estimators_stats — list of {name, month, count}: how many times each SDK estimator was used that month.

• estimators_performance_stats — list of {name, month, batch_size, execution_time}: average batch size and execution time per SDK estimator that month (either can be null if that estimator never reported it).

• image_processing_stats — one entry per month, with image_load_time (average image load execution time) plus three histograms bucketed as {bucket_key: count}: image_size (w_<width>_h_<height>), face_detection_size (h_<height>) and face_descriptor_score (score_<value>).

Where each field comes from

Every field is a downsampled view of the raw requests/errors/SDK-estimator event series described above (Data for monitoring): daily aggregates are pre-computed by the mechanisms in Aggregated statistics, and luna-admin further collapses those daily aggregates into monthly ones at request time.

API field	Aggregation	Source event series	ClickHouse table	InfluxDB measurement / field
request_stats[].count	daily count(), monthly sum	requests	requests_count_daily	route_stats / count
errors[].count	daily count(), monthly sum	errors	errors_statistics_daily	route_errors / count
response_time[].avg	daily avg(execution_time); monthly avgMerge of the daily states on ClickHouse (an exact, count-weighted average over the whole month, not a mean of the daily averages), mean of the daily values on InfluxDB (see note below)	requests	response_time_aggregated_daily	route_stats / avg
response_time[].p95	daily quantile(0.95)(execution_time); monthly quantile(0.95) merge of the daily states on ClickHouse, mean of the daily values on InfluxDB (see note below)	requests	response_time_p95_aggregated_daily	route_stats / p95
response_time[].p99	daily quantile(0.99)(execution_time); monthly quantile(0.99) merge of the daily states on ClickHouse, mean of the daily values on InfluxDB (see note below)	requests	response_time_p99_aggregated_daily	route_stats / p99
estimators_stats[].count	daily sum per estimator, monthly sum	every SDK estimator series (e.g. face_detection, face_descriptor, …)	usages_statistics_daily	aggregate_usages_stats
estimators_performance_stats[].execution_time	daily avg(execution_time), monthly avg	every SDK estimator series	sdk_estimator_execution_time_statistics	sdk_estimator_stats / execution_time
estimators_performance_stats[].batch_size	daily avg(batch_size), monthly avg	every SDK estimator series	sdk_estimator_batch_size_statistics	sdk_estimator_stats / batch_size
image_processing_stats[].image_load_time	daily avg(execution_time), monthly avg	image_load	image_load_time_aggregated_daily	sdk_image_load_stats / avg
image_processing_stats[].image_size	daily count per width/height bucket, monthly sum	image_load	sdk_image_size_statistics_daily	sdk_image_size_stats
image_processing_stats[].face_detection_size	daily count per detected-face-height bucket, monthly sum	face_detection, face_redetection, human_detection	sdk_face_detection_height_statistics	sdk_face_detection_height_stats
image_processing_stats[].face_descriptor_score	daily count per descriptor-score bucket, monthly sum	face_descriptor	face_descriptor_score_statistics	face_descriptor_score_stats

Notes:

• All ClickHouse aggregation tables above key on (date, service, route, status_code) (or (date, estimator, service) for the SDK tables) and use AggregatingMergeTree/SummingMergeTree engines fed by a materialized view off the matching raw series (requests/errors/estimator tables); luna-admin reads the SummingMergeTree tables back with a plain sum and the AggregatingMergeTree ones with avgMerge/quantileMerge as appropriate, and re-groups the daily rows into months. These tables and their materialized views live in the monitoring_db_tools repository’s migrations, not in luna-admin itself.

• On InfluxDB, all of the above (except the histogram-shaped image_processing_stats entries, which are written directly as their bucketed _field) live in the same two aggregated measurements as the raw series they summarize — route_stats (count/avg/p95/p99 fields) and route_errors (count) — written daily by the flux tasks under storages/storages/data/flux/ (collect_response_count.flux, collect_avg_response_time.flux, collect_response_time_p95.flux, collect_response_time_p99.flux, collect_route_errors_statistics.flux, and their SDK/image counterparts), then further downsampled to months by app.utils.query_helpers.fluxSysInfoQuery at request time.

• On InfluxDB, the monthly response_time[].p95/p99 (and, more mildly, avg) are computed by averaging the daily values (aggregateWindow(fn: mean) in app.utils.query_helpers.fluxSysInfoQuery), not by re-deriving a true monthly percentile the way ClickHouse’s quantileMerge does — InfluxDB doesn’t retain the underlying quantile sketches needed for that. So these fields can read slightly differently between the two monitoring backends for the same underlying traffic, especially in months with uneven day-to-day request volume or response-time variance.

Classes

Module contains points for monitoring.

class luna_admin.crutches_on_wheels.cow.monitoring.points.BaseMonitoringPoint(eventTime)

Abstract class for points

eventTime

event time as timestamp

Type:

float

abstract property fields: Dict[str, int | float | str]

Get fields from point. We supposed that fields are not indexing data

Returns:

dict with fields.

abstract property tags: Dict[str, int | float | str]

Get tags from point. We supposed that tags are indexing data

Returns:

dict with tags.

class luna_admin.crutches_on_wheels.cow.monitoring.points.BaseRequestMonitoringPoint(requestId, resource, method, requestTime, service, statusCode)

Base class for point which is associated with requests.

requestId

request id

Type:

str

route

concatenation of a request method and a request resource

Type:

str

service

service name

Type:

str

statusCode

status code of a request response.

Type:

int

property fields: Dict[str, int | float | str]

Get fields

Returns:

“request_id”

Return type:

dict with following keys

property tags: Dict[str, int | float | str]

Get tags

Returns:

“route”, “service”, “status_code”

Return type:

dict with following keys

class luna_admin.crutches_on_wheels.cow.monitoring.points.DataForMonitoring(tags=<factory>, fields=<factory>)

Class fo storing an additional data for monitoring.

class luna_admin.crutches_on_wheels.cow.monitoring.points.InfluxFormatter

Format any point filed into inline format

class luna_admin.crutches_on_wheels.cow.monitoring.points.MonitoringPointInfluxFormatBuilder(name, bases, namespace, /, **kwargs)

Complement point class with explicit fields formatting function for the sake of better performance

To perform type format building target class must have ‘fields’ property return value annotated with TypedDict.

Target class might be configured via ‘Config’ class.

Available options:

extraFields: whether class should handle additional fields or not

>>> from typing import TypedDict
>>>
>>> class MonitoringFields(TypedDict):
...     field1: str
...     field2: int
...     field3: float
...     field4: bool
>>>
>>> class BasePoint(BaseMonitoringPoint, metaclass=MonitoringPointInfluxFormatBuilder):
...
...     def __init__(self, fields: dict):
...         self._fields = fields
...
...     @property
...     def tags(self):
...         return {}
...
...     @property
...     def fields(self) -> MonitoringFields:
...         return self._fields
...
>>> class TestPointNoExtra(BasePoint):
...
...     class Config:
...         extraFields = False
...
>>> class TestPointWithExtra(BasePoint):
...     class Config:
...         extraFields = True
>>>
>>>
>>> point1 = TestPointNoExtra({"field1": "data", "field2": 1, "field3": 1.0, "field4": False})
>>> point2 = TestPointWithExtra({"field1": "data", "field2": 1, "field3": 1.0, "field4": False, "extra": True})
>>> print(point1.convertFieldsToInfluxLineProtocol())
field1="data",field2=1i,field3=1.000000,field4=False
>>> print(point2.convertFieldsToInfluxLineProtocol())
field1="data",field2=1i,field3=1.000000,field4=False,extra=True

classmethod buildInfluxFormats(annotations, extraFields)

Build map with influx formats for corresponding fields

Parameters:

• annotations (dict) – point fields type annotations

• extraFields (bool) – whether point uses extra fields or not

Return type:

dict

Returns:

dict of fields with their format

static convertFieldsToInfluxLineProtocolNoExtra(point)

Convert point fields into influx line protocol format without extra fields

static convertFieldsToInfluxLineProtocolWithExtra(point)

Convert point fields into influx line protocol format with extra fields

static getTypeFormat(_type, _field, extraFields)

Get field type format

Parameters:

• _type (type) – field type

• _field (str) – field name

• extraFields (bool) – whether point uses extra fields or not

Return type:

str

Returns:

string format of the field

class luna_admin.crutches_on_wheels.cow.monitoring.points.RequestErrorMonitoringPoint(requestId, resource, method, errorCode, service, requestTime, statusCode, additionalTags=None, additionalFields=None)

Request monitoring point is suspended for monitoring requests errors (error codes)

errorCode

error code

Type:

int

additionalTags

additional tags which was specified for the request

Type:

dict

additionalFields

additional fields which was specified for the request

Type:

dict

property fields: Dict[str, int | float | str]

Get fields.

Returns:

dict with base fields and additional tags

series: str = 'errors'

series “errors”

property tags: Dict[str, int | float | str]

Get tags.

Returns:

dict with base tags, “error_code” and additional tags

class luna_admin.crutches_on_wheels.cow.monitoring.points.RequestMonitoringPoint(requestId, resource, method, executionTime, requestTime, service, statusCode, additionalTags=None, additionalFields=None)

Request monitoring point is suspended for monitoring all requests and measure a request time etc.

executionTime

execution time

Type:

float

additionalTags

additional tags which was specified for the request

Type:

dict

additionalFields

additional fields which was specified for the request

Type:

dict

property fields: Dict[str, int | float | str]

Get fields.

Returns:

dict with base fields, “execution_time” and additional tags

series: str = 'requests'

series “request”

property tags: Dict[str, int | float | str]

Get tags.

Returns:

dict with base tags and additional tags

luna_admin.crutches_on_wheels.cow.monitoring.points.getRoute(resource, method)

Get a request route, concatenation of a request method and a request resource :type resource: str :param resource: resource :type method: str :param method: method

Returns:

{resource}”

Return type:

str

luna_admin.crutches_on_wheels.cow.monitoring.points.monitorTime(monitoringData, fieldName)

Context manager for timing execution time.

Parameters:

• monitoringData (DataForMonitoring) – container for saving result

• fieldName (str) – field name

Module implement base class for monitoring

class luna_admin.crutches_on_wheels.cow.monitoring.manager.LunaMonitoringManager(settings, name, pluginManager)

Monitoring manager. Sends data to the monitoring storage and monitoring plugins. .. attribute:: settings

monitoring storage settings

async close()

Stop monitoring.

Return type:

None

flushPoints(points)

Flush point to monitoring.

Parameters:

points (Iterable[TypeVar(TPoint, bound= BaseMonitoringPoint)]) – point

Return type:

None

async initialize()

Initialize monitoring

Return type:

None

async probe()

Check that we can connect to the monitoring service with current configuration. Can be used without initialization

class luna_admin.crutches_on_wheels.cow.monitoring.manager.MonitoringSettings(*args, **kwargs)

Monitoring settings protocol

class ClickhouseCredentials

Monitoring credentials

class InfluxCredentials

Monitoring credentials

Module contains classes for sending a data to an influx monitoring.

class luna_admin.crutches_on_wheels.cow.monitoring.influx_adapter.InfluxMonitoringAdapter(settings, flushingPeriod)

Influx 2.x adaptor. Suspended to send points to an influxdb

bucket

influx bucket name

Type:

str

addPointsToBuffer(points)

Add points to buffer.

Parameters:

points (Iterable[BaseMonitoringPoint]) – points

Return type:

None

clearBuffer()

Clear monitoring buffer

Return type:

list[str]

static convertFieldsToInfluxLineProtocol(fields)

Convert field value to influx line protocol format

Parameters:

fields (dict) – dict with values to convert

Return type:

str

Returns:

line protocol string

generatePointStr(point)

Generate string from point

Parameters:

point (BaseMonitoringPoint) – point

Return type:

str

Returns:

influx line protocol string

async stopMonitoring()

Stop monitoring (cancel all request and stop getting new).

Return type:

None

class luna_admin.crutches_on_wheels.cow.monitoring.influx_adapter.InfluxSettings(url, bucket, organization, token)

Container for influx 2.x settings