How to upload files into Google Cloud Storage using NiFi

Setup controller service for GCP

Right click → Configure

Click on +

Choose GCPCredentialsControllerService

Click on Configure

Copy your service account json into this field, refer here on how to set up service account https://cloud.google.com/iam/docs/creating-managing-service-accounts or refer to my video on the detailed step

{
  "type": "service_account",
  "project_id": "abc",
  "private_key_id": "abc",
  "private_key": "abc",
  "client_email": "abc",
  "client_id": "abc",
  "auth_uri": "abc",
  "token_uri": "abc",
  "auth_provider_x509_cert_url": "abc",
  "client_x509_cert_url": "abc"
}

Click on Apply, wait for a while for it to validate. You might need to exit first and come back to the same UI again as I am not sure why the Validating will take some time if you don't do that.

Finally enable it

Write to Google Cloud Storage (GCS)

Create a folder first → then you start dragging the processors

GenerateFlowFIle Settings

PutGCSObject Settings

Choose your project id, the credentials you just setup, bucket name , file name and file type

Finally just start everything and check the file from the GCS UI

This is part of my online course on how to kickstart data engineering workflows in Google Cloud Platform (GCP) for beginners, sign up here to watch detailed video explanation! 🤗