2 min read

How to upload files into Google Cloud Storage using NiFi

Configure NiFi controller services based on GCP service account, create dummy file and write into Google Cloud Storage without any code.
How to upload files into Google Cloud Storage using NiFi
How to upload files into Google Cloud Storage using NiFi

Setup controller service for GCP

Right click → Configure

Configure controller service in NiFi
Configure controller service in NiFi

Click on +

How to add controller service
How to add controller service 

Choose GCPCredentialsControllerService

Choose GCPCredentialsControllerService
Choose GCPCredentialsControllerService

Click on Configure

Configure GCPCredentialsControllerService
Configure GCPCredentialsControllerService

Copy your service account json into this field, refer here on how to set up service account https://cloud.google.com/iam/docs/creating-managing-service-accounts or refer to my video on the detailed step

Input Service Account JSON based on downloaded service account json
Input Service Account JSON based on downloaded service account json
{
  "type": "service_account",
  "project_id": "abc",
  "private_key_id": "abc",
  "private_key": "abc",
  "client_email": "abc",
  "client_id": "abc",
  "auth_uri": "abc",
  "token_uri": "abc",
  "auth_provider_x509_cert_url": "abc",
  "client_x509_cert_url": "abc"
}

Click on Apply, wait for a while for it to validate. You might need to exit first and come back to the same UI again as I am not sure why the Validating will take some time if you don't do that.

How to reduce wait for validation of controller service
How to reduce wait for validation of controller service

Finally enable it

Enable controller service
Enable controller service 

Write to Google Cloud Storage (GCS)

Create a folder first → then you start dragging the processors

High level architecture in NiFi
High level architecture in NiFi

GenerateFlowFIle Settings

Set run schedule to a much larger number to prevent spam
Set run schedule to a much larger number to prevent spam
Input custom text as the content of file
Input custom text as the content of file

PutGCSObject Settings

Choose your project id, the credentials you just setup, bucket name , file name and file type

Sample setting of PutGCSObject
Sample setting of PutGCSObject

Finally just start everything and check the file from the GCS UI


This is part of my online course on how to kickstart data engineering workflows in Google Cloud Platform (GCP) for beginners, sign up here to watch detailed video explanation! 🤗