Example Data Flows using Apache NiFi

Apache NiFi is a stable, high-performance, and flexible platform for building custom data flows. The examples below are a selection of BatchIQ data flow experience using Apache NiFi, Amazon Web Services, Hadoop, and other components.

Process S3 Event Notifications with Apache NiFi

S3 provides a range of design options for processing data. This example dives into handling S3 event notifications, for streaming processing of files immediately after they are written to your S3 bucket.

more...
aws sqs s3

Process S3 Event Notifications with Apache NiFi

S3 provides a range of design options for processing data. This example dives into handling S3 event notifications, for streaming processing of files immediately after they are written to your S3 bucket.

more...
aws sqs s3

Processing CloudTrail Events with NiFi

This example shows how to process a stream of AWS CloudTrail events using Apache NiFi with AWS SQS, S3, and Amazon Athena.

NiFi is a great platform for processing CloudTrail events, with low-latency handling and the ability to both directly process and support a wide variety of downstream options.

more...
aws cloudtrail sqs s3 athena

Processing CloudTrail Events with NiFi

This example shows how to process a stream of AWS CloudTrail events using Apache NiFi with AWS SQS, S3, and Amazon Athena.

NiFi is a great platform for processing CloudTrail events, with low-latency handling and the ability to both directly process and support a wide variety of downstream options.

more...
aws cloudtrail sqs s3 athena

Amazon Athena Ingest with Apache NiFi

Amazon Athena is a recently launched service that provides interactive SQL queries over your data in S3. Athena uses the Hive Metastore to define your data structure, and Presto for processing queries.

Apache NiFi is a great tool for building an ingest pipeline to the Amazon Athena query service, and through Athena to other AWS data tools.

more...
aws s3 athena

Amazon Athena Ingest with Apache NiFi

Amazon Athena is a recently launched service that provides interactive SQL queries over your data in S3. Athena uses the Hive Metastore to define your data structure, and Presto for processing queries.

Apache NiFi is a great tool for building an ingest pipeline to the Amazon Athena query service, and through Athena to other AWS data tools.

more...
aws s3 athena

S3 Ingest with Apache NiFi

This example show how to use an Apache NiFi flow to package and route data to S3 storage. Typically S3 is used as a Data Lake, or an intermediate destination awaiting further pipelines to Redshift or HDFS.

more...
aws s3

S3 Ingest with Apache NiFi

This example show how to use an Apache NiFi flow to package and route data to S3 storage. Typically S3 is used as a Data Lake, or an intermediate destination awaiting further pipelines to Redshift or HDFS.

more...
aws s3

Database Extract with NiFi

Apache NiFi has built-in processors for extracting database data into NiFi FlowFiles, and this example looks at the pros and cons of each for building a database flow.

more...
database

Database Extract with NiFi

Apache NiFi has built-in processors for extracting database data into NiFi FlowFiles, and this example looks at the pros and cons of each for building a database flow.

more...
database

Database Ingest with NiFi

Apache NiFi is a great tool for building flexible and performant data ingestion pipelines. Since relational databases are a staple for many data cleaning, storage, and reporting applications, it makes sense to use NiFi as an ingestion tool for MySQL, SQL Server, Postgres, Oracle, etc.

more...
database

Database Ingest with NiFi

Apache NiFi is a great tool for building flexible and performant data ingestion pipelines. Since relational databases are a staple for many data cleaning, storage, and reporting applications, it makes sense to use NiFi as an ingestion tool for MySQL, SQL Server, Postgres, Oracle, etc.

more...
database

Using Apache NiFi for Elastic MapReduce Ingest

Amazon Elastic MapReduce (EMR) is a great managed Hadoop offering that allows clusters to be both easily deployed and easily disolved. EMR can be used to set up long-lived clusters or run scripted jobs priced by the hour. This example covers the basics for connecting Apache NiFi to an EMR cluster.

more...
aws emr hadoop hdfs

Using Apache NiFi for Elastic MapReduce Ingest

Amazon Elastic MapReduce (EMR) is a great managed Hadoop offering that allows clusters to be both easily deployed and easily disolved. EMR can be used to set up long-lived clusters or run scripted jobs priced by the hour. This example covers the basics for connecting Apache NiFi to an EMR cluster.

more...
aws emr hadoop hdfs