How to install NiFi container in Google Cloud Platform

Steps

Create a new project in GCP (this is to simulate your local environment or on premise infrastructure) 😎

Set up firewall rule to allow nifi port 9090

Search firewall → Create Firewall Rule

Set up VM → Search VM instances

• Create a new VM instances, name it as anything that you like (e.g. local-machine)
• Choose us-central1 to have the lowest cost
• Choose n2-standard-2 for our demo purposes, it will cost about 40-50 USD per month, even if you turn on 24/7! Just switch it off when you don't use it so you can use your free trial on other services.
• Boot disk → just leave it as default which is Debian GNU/Linux 10 (buster)

For Identify and API access, choose No Service Account since we do not require any special permission attach to the VM

Firewall → Allow HTTP traffic, Allow HTTPS traffic

Click on Management, security, disks, networking, sole tenancy, Choose Networking, add the network tags you created on the above step e.g. nifi

Leave everything as default and click on Create

Ssh into the VM

Install docker on VM, reference https://docs.docker.com/engine/install/debian/

# Update the apt package index and install packages to allow apt to use a repository over HTTPS:
sudo apt-get update
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

# Add Docker’s official GPG key:
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# Set up the stable repository
echo \
  "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install the latest docker engine
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io


# Add your user to docker and activate group 
# Replace user_xyz with your username, can use command "whoami" to check your username 
whoami
sudo usermod -aG docker user_xyz
newgrp docker

# You should be able to run this command 
docker ps

Install nifi, refer here for more details https://hub.docker.com/r/apache/nifi/, replace the password with your own password

# Setting HTTP port as 9090 and downloading nifi 1.13 for stability purpose
docker run --name nifi \
-p 9090:9090 \
-d \
-e NIFI_WEB_HTTP_PORT='9090' \
apache/nifi:1.13.0

(Optional) check logs just in case there is any issue

# Run docker ps to get the nifi container id 
docker ps

# Ssh into the container and check the logs 
docker exec -it (containerid) /bin/bash

tail -f logs/nifi-app.log

# exit container
ctrl+d

Access nifi based on the external ip

Do not share your external IP with anyone!!

Access the website

http://33.100.100.100:9090/nifi

This is part of my online course on how to kickstart data engineering workflows in Google Cloud Platform (GCP) for beginners, sign up here to watch detailed video explanation! 🤗