Apache Nifi
[wtm_mlop_cats]
Put simply, NiFi was built to automate the flow of data between systems. While the term ‘dataflow’ is used in a variety of contexts, we use it here to mean the automated and managed flow of information between systems. This problem space has been around ever since enterprises had more than one system, where some of the systems created data and some of the systems consumed data. The problems and solution patterns that emerged have been discussed and articulated extensively.
Features
Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of Apache NiFi include:
1. Web-based user interface
Seamless experience between design, control, feedback, and monitoring
2. Highly configurable
Loss tolerant vs guaranteed delivery
Low latency vs high throughput
Dynamic prioritization
Flow can be modified at runtime
Back pressure
3. Data Provenance
Track dataflow from beginning to end
4. Designed for extension
Build your own processors and more
Enables rapid development and effective testing
5. Secure
SSL, SSH, HTTPS, encrypted content, etc…
Multi-tenant authorization and internal authorization/policy management