Integrating your data teams with NiFi: a step-by-step guide 

Integrating your data teams with NiFi: a step-by-step guide 

NiFi is an easy-to-use, powerful, and reliable system to process and distribute data. So, how can we consolidate our organization’s data team using this one specific tool?

We recently shared our perspective on how to build a data team. Because we need to always try to simplify the process for them to work together, here is an option we can implement by using only one tool. 

Just to clarify, there are thousands of data tools that can be synced together to make your data platform. Matt Turck already gathered a lot of information for us with his Data Landscape, but we are going to focus on the use of one tool, in order to guide you into reducing your data stack. 

Data teams 

In our past article we have already mentioned the skills you should be looking for when hiring, and hopefully your team looks like this now: 

  • Data Engineers 
  • DataOps 
  • Data analysts 
  • Data Stewards 

In our experience, we have noticed that by using Apache NiFi, you can work with your data on multiple levels and, of course, teams. So, let’s get into it. 

What is NiFi? 

NiFi is “An easy to use, powerful, and reliable system to process and distribute data.” and has also been called by Cloudera as a “Low code streaming data processing”, which means, you can build Data Flows with very little effort, but, in the era of video, you can see it for yourself

NiFi

High-level features in NiFi are: 

  • Web-based user interface 
  • Highly configurable 
  • Data provenance 
  • Designed for extension 
  • Secure 

Features we have enjoyed while working with it: 

  • Streamlined work 
  • Easy to support 
  • Easy to build one-time off solutions 
  • Easy to run locally 
  • Granular access 
  • Scalable 
  • Resilient 
  • On-prem to Cloud easy integration 

How will it help integrate your teams? 

Now that we have a glimpse of what NiFi is, we want to focus on two teams that can work together on top of NiFi, Data Engineers and DataOps. 

NiFi for Data Engineers 

In today’s world a Data Engineer has a wide range of responsibilities on the Data Stack, from building the platform to creating the custom pipelines for that specific business needs that just came up, this role is very often randomized. 

We understand that randomization is not something we want, it just happens in every organization because of “Time to Market”. This is the reason why we need to speed up development for those moments where we want data right away. 

In this sample, we can see a flow that was built in about 2 minutes (just because we had to get secret keys from Twitter). These are the steps it follows: 

  • Listen to twitter for a specific hashtag 
  • Send it to S3 

This is, of course, a very simple flow. It could get way more complex than this but we are not talking about all that NiFi can do, we are interested in how we can allow teams to work together using this app. 

As you can see data is already flowing and we are storing tweets based on a hashtag we decided to follow, into the bucket we wanted. But the important part is we can now ship this flow to prod and hand it over to DataOps Engineers to make sure it is working, or change it as required. 

In summary, the Data Engineer randomization took 2 minutes of their time to deliver the tweets we have to analyze. 

NiFi for DataOps 

Based on the example above, what can DataOps do? Let’s see some examples: 

  • Change the hashtag to follow 
  • Change the bucket to put tweets in 
  • Re-post tweets in case they are deleted 

So yes, this tool gives DataOps some level of freedom. But what happens when you have hundreds of flows running in your NiFi Instance, this is how it looks: 

It is impossible to manage that, NiFi’s canvas is just too convoluted by having all flows in it. You can be more organized, add ways to keep things simple and apply every best practice, but the canvas will keep growing every time you have planned or randomized flows created.  

Of course, the NiFi team already thought about it, and added a feature for DataOps (or support teams) who need to locate a single flow quickly or simply monitor what’s going on in that instance. 

As you can see in the top right corner there is a menu icon that holds some interesting options in NiFi: 

I just want to emphasize a few of them that will help DataOps teams handle NiFi easily: 

  • Summary: Shows the canvas but in a list format, either per processor (like GetTwitter) or per entire flow (Process Group). It has a filter where you can just put the name and then click on the right arrow icon on the right side to get redirected inside of the flow. 
  • Bulletin board: Shows the error log, so in case of failure you can get there and just see what’s going on in the entire NiFi instance. 
  • Data Provenance: A centralized place where every change applied to the data is recorded. It keeps track and stores every event that occurs in the flow like getting file, selecting few columns, converting to parquet, among others. Everything gets in there with the same right arrow icon that will drive you to it. This feature is at processor level. 
  • Flow configuration history: This is a record of everything that a user has done in NiFi (except for login in). So, if a user starts, stops, disables, changes the configuration or creates a new flow, it will be registered here with the user’s identifier and every step done. 

Conclusion 

At this point, I think you may be looking into, at least, having a NiFi PoC in your stack. And this is just one of the possible applications of NiFi to consolidate your organization’s data team. 

Keep in mind NiFi currently (as of version 1.16.2) has 309 pre-built processors that allow you to: 

  • Use regular databases, such as SQL Server, MySQL, Postgres, etc. 
  • Use no-SQL databases like MongoDB or Elasticsearch 
  • Use “Big Data” ecosystems with Hadoop, Hive, Presto, Spark, Kafka, etc. 
  • Interact with multiple cloud services in AWS, Azure and GCP 
  • Convert data into multiple data formats csv, json, parquet, avro, etc. 
  • Act as a socket for message handling 
  • Act as an API handler 
  • Perform predefined web scraping 
  • Receive and send emails 

So, does this sound good to you? If the answer is yes, then you can download it and try it out from here: https://nifi.apache.org/ 

Comments?  Contact us for more information. We’ll quickly get back to you with the information you need. 

WHY TO HAVE IT?

DevOps is a new way of collaboration that makes processes change for the better. The idea of this paper is to understand why you need DevOps and to learn how you can get started. You will walk through the key ideas that bring development and operations together to produce higher-quality software.

White Paper
×

WHY TO HAVE IT?

With an eye on business demands on having clean code, the idea of Technical Debt has emerged. The problem is a software product can only handle so much Technical Debt before it begins to crumble under the weight of its own inefficiency. Learn here how to tackle Technical Debt properly.

White Paper
×

WHY TO HAVE IT?

Companies who are successfully incorporating continuous innovation into their business practices have changed the way their enterprise software is deployed. Trends in application development and deployment support organization-wide transformation but which type of architecture is right for you?

White Paper
×

READY TO TAKE A LOOK?

Many organizations around the world are transferring their IT and software development operations to third parties. For most of them, thinking about outsourcing is an issue of efficiency and focus. Don’t miss these interesting facts that show you why this might be the perfect choice for your business.

White Paper
×

WHY TO HAVE IT?

In the search for the perfect solution, it is not uncommon to find that off-the-shelf products don’t offer solutions fitting your needs and business model. This is where a software development partner comes in. If you are looking to understand what software consultants companies can do for you, keep reading.

White Paper
×

Event-Driven Architecture (EDA) An old concept for a modern context

The concept of Event-Driven Architecture (EDA) may not be new in the software development industry; however, it is becoming stronger once again, thanks to its flexibility and adaptability by using information in real time. At the same time, EDA offers companies the opportunity to process events and flows of events that result in immediate analysis.

White Paper
×

A Beginner’s Guide to Scrum in Agile Projects

Scrum has proven to be very useful in any field and industry, regardless of its complexity. This quick and easy guide aims to show the main insights this methodology has, how it works, and other important aspects to consider if you want to start in the Agile world and improve your company’s results.

White Paper
×