Luigi
[wtm_mlop_cats]
The purpose of Luigi is to address all the plumbing typically associated with long-running batch processes. You want to chain many tasks, automate them, and failures will happen. These tasks can be anything, but are typically long running things like Hadoop jobs, dumping data to/from databases, running machine learning algorithms, or anything else.
Features
Luigi has features such as visualiser page, dependency graph, task, scheduler, explicit dependencies between workflows, easier to write, vastly more extensible, code reuse and can easily re-run individual steps.