James Wing By James Wing on 2016-03-30

This article describes how to upgrade to a newer version of Apache NiFi.

Planning an Upgrade

Apache NiFi is an exciting and dynamic project undergoing active development, and there are always new features worth upgrading to. But upgrading can be disruptive to your flow processing without careful planning and execution. The best way to upgrade is to migrate your data flow to a new NiFi instance running a newer version of NiFi. This ensures that you have not only the latest version of Apache NiFi, in its ideal configuration, but also the latest BatchIQ image definition with upgrades for security, monitoring, and deployment. It also preserves your existing instance as a configuration benchmark and rollback option. Some things to consider:

  • Flow defintition - You will need to make your flow portable, by saving a copy of the flow definition as a NiFi Template to migrate to the new server.
  • Configuration and settings - Make sure you identify and migrate any custom settings on the instance, in nifi.properties, in security configuration, or anything on the instance but outside the flow.
  • Transition planning - Consider how your flow can be safely run and monitored on multiple servers at the same time. Also, you should plan on retaining your old server active for long enough to "drain" in-process flow files, and for as long as you would like to review provenance data or logs.

Saving your Flow Template

You will need to export your existing NiFi flow as a template, so it can be copied to the new instance.

Starting a New Instance

Set up a new instance running Apache NiFi. By importing your flow template, you can replicate your existing flow behavior as long as you make sure the following are true:

  • The instance has the same IAM role permissions for accessing resources in your AWS account like S3, CloudWatch Logs, etc.
  • Update any sensitive properties, like credentials, on Processors and Controller Services. Sensitive properties are not included in NiFi templates.
  • Instance configurations such as nifi.properties entries, security settings, and .NAR files are copied over.
  • Look for version migration warnings that might come up from new Processor settings or behaviors in the new NiFi version.

Transitioning Flows

At this point, you should have both the old and the new NiFi servers running, but the NiFi flow only running on the old server. You will now transition the running flow from the old instance to the new instance.

  1. Stop input Processors in the NiFi flow on the old instance, but allow flow files to continue completing the flows, to "drain" live flow files from the old instance
  2. Start the NiFi flow on the new instance
  3. Monitor both instances to ensure the flow has transitioned correctly to the new instance

Closing Out the Old Instance

Do not immediately terminate your old instance. You may wish to keep it while you verify correct behavior on the new instance, review stored provenance data, or access local content files. Stopping an EC2 instance without Terminating can be a cost-effective way to maintain an old instance for a while.

Need Some Help?

Please use the BatchIQ Support Portal to open a ticket and get customized assistance.