How It Works


how

how-it-works

In a nutshell.

The task life cycle could be summarized in the following steps:

  • A task is submitted via an HTTP API
  • The task will be stored in a database (MySQL) for keeping track of it
  • Based on the schedule, when is time to run it will be moved to a Redis queue
  • The task is dispatched via gRPC to the targets
  • The status of the task will be updated in the database

The process is pretty straightforward one of the main design goals is to avoid unnecessary complexity, giving priority to performance and simple orchestration of the tasks at all cost as for the maintenance and scalability of the full system.

Going further and in a try to extend this simplicity for 3rth party services using marabunta as a service, a notification service is exposed in order to return notification in real time either via HTTP or gRPC allowing with this applications to integrate or take actions based on the status of a task and receive instant notifications instead of long polling for example.

Below, more information in detail about the flow in each stage, also check the prototype

HTTP REST API

All interaction from the user side is via an HTTP REST API using JSON as the format, from the creation of the tasks as for query status about it.

The tasks can be:

  • on-demand
  • scheduled

The on-demand tasks are in a first come, first served basis, the scheduled follow a CRON expression like pattern.

A draft of the JSON payload for creating a task on-demand could be something like:

{
    "name": "Hello World",
    "target": "all",
    "when": "now",
    "payload": {
        "task": "echo hi!"
    }
}

A draft of the JSON payload for creating an scheduled task could be something like:

{
    "name": "Hello World",
    "target": "all",
    "when": "@every 1h30m10s",
    "payload": {
        "task": "echo I run every 1h, 30m and 10s from my creation time"
    }
}

or

{
    "name": "Hello World",
    "target": "all",
    "when": "* */2 * * *",
    "payload": {
        "task": "echo I run every 2 hours based on the server time"
    }
}

Check the payload section for more details.

From HTTP to MySQL

Before adding a task into the DB, the payload is verified, the main check is against the type of the task (value when).

If the task is to be done “NOW”, the task asynchronously is also queued into Redis if not later will be picked up and dispatched according to its schedule.

From MySQL to Redis

The resolution for scheduled tasks is of 1 minute, this means that every minute a database query is run to fetch and check for new tasks to be dispatched.

One or more instances (when scaling) could read the database and enqueue later into Redis, to prevent possible race conditions or duplicates the queries are transactional (InnoDB) from the DB size (WHERE state == "todo").

In the Redis side the to-do queue is a sorted set with the following format:

marabunta:todo <timestamp of the task> <task UUID>

The timestamp of the task is obtained from the defined scheduled same as the task UUID, these two values are unique and consistent helping to guarantee that if one or more “workers” read from the SQL database and write to the Redis “sorted set” at the same time, by design the task record will be only updated but not duplicated, also the resolution for the dispatcher in the Redis side is 1 second (minimum unit on time on the Unix timestamp) in other words this helps to do an atomic transaction.

Dispatching the task

The resolution for dispatching scheduled tasks is 1 second, but this doesn’t mean that all the tasks need to wait one second before being executed, the on-demand tasks skip the “todo” queue and are “queued” directly to be processed.

As described before, one or more processes query the database (MySQL) and then enqueue the tasks into Redis, the dispatching stage works exactly the same but in this case within Redis. To achieve “atomic transactions” when having more than one client “dispatching” the use of the command WATCH is used, more can be found in the Redis docs https://redis.io/topics/transactions.

The dispatcher query the marabunta:todo order set by using, something like:

 ZRANGEBYSCORE marabunta:todo 0, <current timestamp> "limit", 0, 1

Only one task is fetched at a time, and later pushed to a list:

 LPUSH marabunta:queued <task UUID>

And then the task is removed from the todo queue:

 ZREM marabunta:todo  <task UUID>

All this runs within a single transaction, by only fetching one record at a time, it may look like a penalty for performance but by doing some basic benchmarks running this on a system having 2.9 GHz Intel Core i9 these were the results for a single “worker”

  • for 100K (100,000) an average of 30 seconds
  • for 1M (1,000,000) an average of 133 seconds less than 3 minutes (25MB disk dump.rdb size)

Spawning more workers to concurrently dequeue, didn’t help much in terms of speed, but is nice to know that if one worker goes down the others take over and no collisions/duplication of tasks occur so no worries when scaling.

Everything is not about speed, but by mixing different technologies together and using them in where they shine must, the outcome is very positive.

gRPC bidirectional stream

Once the tasks arrive into the marabunta:queued, the redis command BRPOPLPUSH is used to deque and process the task UUID.

marabunta keeps record of their available clients (🐜) and will send the payload only to the targets defined in the task and that are available.

Once the task execution finished, using the same channel the status is sent back and status is updated on the database.

comments powered by Disqus