There are many good tools, frameworks offering multiple solutions for managing jobs, tasks, queues, etc, many of them are listed here http://queues.io/.
Technology changes and with this, new tools emerge. Days back you could be using gearman, these days probably nats.io or even ansible ad-hoc:
$ ansible all -a "ping -c3 184.108.40.206" -u devops --become
Just as an example that with multiple technologies the same result can be achieved, some relatively easy, others faster, etc. it all depends on what are needs and problem to solve.
marabunta design principle could be seen as the glue between multiple technologies in the aim to achieve a balanced distributed task manager, is not trying to reinventing the wheel but rather working more like the bridge between technologies in order to offer a light, stable and secure system following best practices.
One main goal is to have a service easy to operate and manage but at the same time have the capacity to scale on demand while keeping a very small footprint in terms of resources in the client side (targeting IoT devices)
Having to backup one, two or 10 systems is doable, but whey you need to backup hundreds and change frequently or temporally the credentials, bucket names etc things become complicated to manage.
One of the goals is to dynamically take advantage of the task
payload in where the database credential could be passed, same as S3 access key etc, simplifying and securing with this the backup process at any scale.
For example, imagine that you need to test a new S3 bucket that supports encryption on REST, therefore you have new access keys and bucket name, to create a basic test the task
payload would contain this new values and that should be enough if your test ran successfully you could simply upload the existing payload for other tasks and that should be enough without even need to access the remote system via ssh for example.
The client has native support for S3 simplifying with this the usage of the cloud storage in use.
Single bidirectional secure connection (client-server)
Many nodes are behind firewalls
block in, pass out once the client establishes a connection with the server it should be available to reuse the connection besides securing it using TLS.
The server will never try to open a connection to a client and communication always should begin from the client to the server.
Kanban for infrastructure
When creating a task for one, two or hundreds of devices (IoT) a simple board representing the progress of the task among all the devices gives a better overview of how things are performing, this can be later extended via solutions like in time series but in the goal to keep things simple with random green dots:
task | to do | in progress | done | ------+-------+-------------+------+ node1 | * | | | node2 | | * | | .... nodeN | | | * |
When using as a service or when some custom actions are required, get a notification based on the events.