An Opined Monitoring And Self Healing System

Plugins Based

Like other frameworks, Lifeguard was built with mechanisms that facilitate the creation of plugins, separating what is essential from what is optional. Example: The core must know when or not to notify, but we don't need to load the MS Teams notification routines if we are going to use Google Chat.

Proactive Validations

Most monitoring systems work reactively, like the Grafana Alertmanager solution, that is, based on things (in many cases, errors) that have already happened that alerts are triggered. Lifeguard's focus is on making it easy to create proactive validations that can look for errors and send easily understandable alerts.

Starting a project

To start a project, you need to install the Lifeguard package using pip:

pip install lifeguard

Create a new directory and create a main settings file used for Lifeguard:

mkdir myproject
cd myproject
lifeguard -g

The -g parameter will create a new file called lifeguard_settings.py with the initial structure. The example in the right side is a example.

# import lifeguard plugin for MongoDB
# This plugin able the persistent results of validations
# To use this plugin, you need to install it using pip
# pip install lifeguard-mongodb
import lifeguard_mongodb

from lifeguard.settings import SettingsManager
from lifeguard.auth import BASIC_AUTH_METHOD

# list of plugins used by this instance
PLUGINS = [lifeguard_mongodb]

def setup(lifeguard_context):
    # Register user and password for basic authentication
    lifeguard_context.auth_method = BASIC_AUTH_METHOD
    lifeguard_context.users = [
        {"username": "user", "password": "pass"}
    ]

Simple Example Validation

from lifeguard import NORMAL, PROBLEM, change_status
from lifeguard.actions.database import save_result_into_database
from lifeguard.http_client import get
from lifeguard.logger import lifeguard_logger as logger
from lifeguard.validations import ValidationResponse, validation


# The validation decorator is used to register a new validation.
# The schedule parameter is used to define when the validation will be executed.
# The actions parameter is used to define what actions will be
# executed after the validation.
# The validation function must return a ValidationResponse object.
# The file with validation must be in the folder validations and
# must be named with "_validation.py" suffix.
@validation(
    "check if pudim is alive",
    actions=[save_result_into_database],
    schedule={"every": {"minutes": 1}},
)
def pudim_is_alive():
    status = NORMAL
    result = requests.get("http://pudim.com.br")
    logger.info("pudim status code: %s", result.status_code)

    if result.status_code != 200:
        status = change_status(status, PROBLEM)

    return ValidationResponse(
        status,
        {status: result.status_code},
    )

lifeguard

lifeguard -d

In the output used as an example (sent to a Telegram channel) the pod was in crash loop and the root cause explanation was generated by the OpenAI API.

In the next example, the first pod has an invalid image and the pull is failing. The second pod has an invalid command to start the container. Both explanations were generated based on events in the pods.

MongoDB

This plugin is used to store the results of validations in a MongoDB database.

TinyDB

This plugin is used to store the results of validations in a TinyDB database. TinyDB produces a single json file with data.

Google Chat Notification

This plugin is used to send notifications to Google Chat.

MS Teams Notification

This plugin is used to send notifications to MS Teams.

Telegram

This plugin is used to send notifications to Telegram.

The plugin provides a simple way to integrate bot commands with Lifeguard.