Monitoring

Data for monitoring

We are currently processing several types of events for monitoring:

1. request (any http request)

2. error (failed http request)

InfluxDB: Every event is a point in the time series. The point is represented as union of the following data:

• series name

• start event time

• tags, indexed data in storage, dictionary: keys - string tag names, values - string, integer, float

• fields, non indexed data in storage, dictionary: keys - string tag names, values - string, integer, float

Clickhouse: In Clickhouse, the data structure resembles that of a traditional SQL table. Each event is represented as a record, where:

• The `time` field contains the record’s creation timestamp;

• The `data` field contains a JSON object with all the information that would otherwise be distributed across tags and fields in InfluxDB.

Important: In Clickhouse, there is no differentiation between “tags” and “fields”—all data is consolidated into a single JSON object within the data field.

Monitoring series

The structure and the meaning of each monitoring series remain consistent. However, for Clickhouse, data from tags and fields are merged into a single JSON object under the data field. Below are examples for each series:

• Requests series.

  Triggered on every request. Each point contains a data about corresponding request (execution time and etc).

  InfluxDB:

  • tags

    tag name	description
    service	always “luna-serving”
    route	concatenation of a request method and a request resource (GET:/version)
    status_code	http status code of response

  • fields

    fields	description
    request_id	request id
    execution_time	request execution time

  ClickHouse JSON `data` field Example:

  {
      "service": "luna-serving",
      "route": "GET:/version",
      "status_code": 200,
      "request_id": "1536751345,6a5c2191-3e9b-f5a4-fc45-3abf43625c5f",
      "execution_time": 123.45
  }
  

• Errors series.

  Triggered on failed request. Each point contains error_code of luna error.

  InfluxDB:

  • tags

    tag name	description
    service	always “luna-serving”
    route	concatenation of a request method and a request resource (GET:/version)
    status_code	http status code of response
    error_code	luna error code

  • fields

    fields	description
    request_id	request id

  ClickHouse JSON `data` field Example:

  {
      "service": "luna-serving",
      "route": "GET:/version",
      "status_code": 400,
      "error_code": 13037,
      "request_id": "1536751345,6a5c2191-3e9b-f5a4-fc45-3abf43625c5f",
  }
  

Database

You can refer to documentation for influx database and clickhouse database to compare the databases and choose what benefit your needs more. Note that clickhouse might be the better choice for aggregation You can setup your database credentials in configuration file in section “monitoring”.

Plugins

You can implement your own plugin for sending monitoring data. See plugins

Module request monitoring plugin example

class crutches_on_wheels.cow.plugins.plugin_examples.request_monitoring_plugin_example.BaseRequestMonitoringPlugin(app)

Base class for requests monitoring.

abstractmethod async flushPointToMonitoring(point, logger)

All plugins must realize this method.

This function call after end of request

Parameters:

• point (TypeVar(MonitoringPoint)) – point for monitoring

• logger – logger

Return type:

None

async handleEvent(points, logger)

Handle event

Parameters:

• *args – positional arg for event handler function

• **kwargs – named arg for event handler function

class crutches_on_wheels.cow.plugins.plugin_examples.request_monitoring_plugin_example.RequestMonitoringPlugin(app)

Example plugin sends a request data for monitoring to third-party source. Only one instance of this class exist during the program execution.

async close()

Stop plugin.

Close all open connections and ect

async flushPointToMonitoring(point, logger)

Callback for sending a request monitoring data.

Parameters:

• point (TypeVar(MonitoringPoint)) – point for monitoring

• logger – logger

Return type:

None

async initialize()

Initialize plugin.

Close all open connections and ect