The very first step is to configure the pipeline configuration. We have to set what machine type the pipeline will use, in which available region, the pipeline will execute and so on.<\/p>\n

We can program our pipeline in JAVA or Python. First, we have to set up the Dataflow Pipeline Options object where we will define the configuration of our pipeline.<\/p>\n

We have used Direct Runner to execute and test the pipeline locally.<\/p>\n

options.setRunner(DirectRunner.class);<\/pre>\n
Once we test it locally, then we can replace Direct Runner with Dataflow Runner. That\u2019s all that we need to deploy our pipeline on Cloud Dataflow.<\/p>\n
options.setRunner(DataflowRunner.class);<\/pre>\n
Apart from this, we also need to pass other configurations too to the pipeline like project id, max number of worker nodes, temp location, staging location, worker machine type, region<\/i> where our pipeline will be deployed, etc.<\/p>\n
Create Pipeline<\/h3>\n
After passing all the configurations to the Dataflow Pipeline Options object, then we will create our Pipeline object.<\/p>\n
Refer below snippet to take a closer look at it.<\/p>\n
public class StorageToBQBatchPipeline {\r\n\r\n  public static void main(String[] args) {\r\n\r\n    \/*\r\n     * Initialize Pipeline Configurations\r\n     *\/\r\n    DataflowPipelineOptions options = PipelineOptionsFactory\r\n        .as(DataflowPipelineOptions.class);\r\n    options.setRunner(DirectRunner.class);\r\n    options.setProject(\"\");\r\n    options.setStreaming(true);\r\n    options.setTempLocation(\"\"); \r\n    options.setStagingLocation(\"\");\r\n    options.setRegion(\"\");\r\n    options.setMaxNumWorkers(1);\r\n    options.setWorkerMachineType(\"n1-standard-1\");\r\n\r\n    Pipeline pipeline = Pipeline.create(options);<\/pre>\n
Like what you are reading? You would like our free live webinars too. Sign up and get notified when we host webinars =><\/span>