Luigi

GitHub Support CommunityData Processing

The purpose of Luigi is to address all the plumbing typically associated with long-running batch processes. You want to chain many tasks, automate them, and failures will happen. These tasks can be anything, but are typically long running things like Hadoop jobs, dumping data to/from databases, running machine learning algorithms, or anything else.

Features

Luigi has features such as visualiser page, dependency graph, task, scheduler, explicit dependencies between workflows, easier to write, vastly more extensible, code reuse and can easily re-run individual steps.

Official website

Tutorial and documentation

Enter your contact information to continue reading