Module dataflow
This page documents the language specification for the gcp package. If you're looking for help working with the inputs, outputs, or functions of gcp resources in a Pulumi program, please see the resource documentation for examples and API reference.
Resources
Others
Resources
Resource Job
class Job extends CustomResourceCreates a job on Dataflow, which is an implementation of Apache Beam running on Google Compute Engine. For more information see the official documentation for Beam and Dataflow.
Example Usage
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
const bigDataJob = new gcp.dataflow.Job("bigDataJob", {
parameters: {
baz: "qux",
foo: "bar",
},
tempGcsLocation: "gs://my-bucket/tmp_dir",
templateGcsPath: "gs://my-bucket/templates/template_file",
});Note on “destroy” / “apply”
There are many types of Dataflow jobs. Some Dataflow jobs run constantly, getting new data from (e.g.) a GCS bucket, and outputting data continuously. Some jobs process a set amount of data then terminate. All jobs can fail while running due to programming errors or other issues. In this way, Dataflow jobs are different from most other Google resources.
The Dataflow resource is considered ‘existing’ while it is in a nonterminal state. If it reaches a terminal state (e.g. ‘FAILED’, ‘COMPLETE’, ‘CANCELLED’), it will be recreated on the next ‘apply’. This is as expected for jobs which run continuously, but may surprise users who use this resource for other kinds of Dataflow jobs.
A Dataflow job which is ‘destroyed’ may be “cancelled” or “drained”. If “cancelled”, the job terminates - any data written remains where it is, but no new data will be processed. If “drained”, no new data will enter the pipeline, but any data currently in the pipeline will finish being processed. The default is “cancelled”, but if a user sets onDelete to "drain" in the configuration, you may experience a long wait for your pulumi destroy to complete.
constructor
new Job(name: string, args: JobArgs, opts?: pulumi.CustomResourceOptions)Create a Job resource with the given unique name, arguments, and options.
nameThe unique name of the resource.argsThe arguments to use to populate this resource's properties.optsA bag of options that control this resource's behavior.
method get
public static get(name: string, id: pulumi.Input<pulumi.ID>, state?: JobState, opts?: pulumi.CustomResourceOptions): JobGet an existing Job resource’s state with the given name, ID, and optional extra properties used to qualify the lookup.
method getProvider
getProvider(moduleMember: string): ProviderResource | undefinedmethod isInstance
public static isInstance(obj: any): obj is JobReturns true if the given object is an instance of Job. This is designed to work even when multiple copies of the Pulumi SDK have been loaded into the same process.
property additionalExperiments
public additionalExperiments: pulumi.Output<string[] | undefined>;List of experiments that should be used by the job. An example value is ["enableStackdriverAgentMetrics"].
property id
id: Output<ID>;id is the provider-assigned unique ID for this managed resource. It is set during deployments and may be missing (undefined) during planning phases.
property ipConfiguration
public ipConfiguration: pulumi.Output<string | undefined>;The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE".
property jobId
public jobId: pulumi.Output<string>;The unique ID of this job.
property labels
public labels: pulumi.Output<{[key: string]: any} | undefined>;User labels to be specified for the job. Keys and values should follow the restrictions
specified in the labeling restrictions page.
NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided.
Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.
property machineType
public machineType: pulumi.Output<string | undefined>;The machine type to use for the job.
property maxWorkers
public maxWorkers: pulumi.Output<number | undefined>;The number of workers permitted to work on the job. More workers may improve processing speed at additional cost.
property name
public name: pulumi.Output<string>;A unique name for the resource, required by Dataflow.
property network
public network: pulumi.Output<string | undefined>;The network to which VMs will be assigned. If it is not provided, “default” will be used.
property onDelete
public onDelete: pulumi.Output<string | undefined>;One of “drain” or “cancel”. Specifies behavior of deletion during pulumi destroy. See above note.
property parameters
public parameters: pulumi.Output<{[key: string]: any} | undefined>;Key/Value pairs to be passed to the Dataflow job (as used in the template).
property project
public project: pulumi.Output<string>;The project in which the resource belongs. If it is not provided, the provider project is used.
property region
public region: pulumi.Output<string | undefined>;property serviceAccountEmail
public serviceAccountEmail: pulumi.Output<string | undefined>;The Service Account email used to create the job.
property state
public state: pulumi.Output<string>;The current state of the resource, selected from the JobState enum
property subnetwork
public subnetwork: pulumi.Output<string | undefined>;The subnetwork to which VMs will be assigned. Should be of the form “regions/REGION/subnetworks/SUBNETWORK”.
property tempGcsLocation
public tempGcsLocation: pulumi.Output<string>;A writeable location on GCS for the Dataflow job to dump its temporary data.
property templateGcsPath
public templateGcsPath: pulumi.Output<string>;The GCS path to the Dataflow job template.
property type
public type: pulumi.Output<string>;The type of this job, selected from the JobType enum
property urn
urn: Output<URN>;urn is the stable logical URN used to distinctly address a resource, both before and after deployments.
property zone
public zone: pulumi.Output<string | undefined>;The zone in which the created job should run. If it is not provided, the provider zone is used.
Others
interface JobArgs
interface JobArgsThe set of arguments for constructing a Job resource.
property additionalExperiments
additionalExperiments?: pulumi.Input<pulumi.Input<string>[]>;List of experiments that should be used by the job. An example value is ["enableStackdriverAgentMetrics"].
property ipConfiguration
ipConfiguration?: pulumi.Input<string>;The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE".
property labels
labels?: pulumi.Input<{[key: string]: any}>;User labels to be specified for the job. Keys and values should follow the restrictions
specified in the labeling restrictions page.
NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided.
Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.
property machineType
machineType?: pulumi.Input<string>;The machine type to use for the job.
property maxWorkers
maxWorkers?: pulumi.Input<number>;The number of workers permitted to work on the job. More workers may improve processing speed at additional cost.
property name
name?: pulumi.Input<string>;A unique name for the resource, required by Dataflow.
property network
network?: pulumi.Input<string>;The network to which VMs will be assigned. If it is not provided, “default” will be used.
property onDelete
onDelete?: pulumi.Input<string>;One of “drain” or “cancel”. Specifies behavior of deletion during pulumi destroy. See above note.
property parameters
parameters?: pulumi.Input<{[key: string]: any}>;Key/Value pairs to be passed to the Dataflow job (as used in the template).
property project
project?: pulumi.Input<string>;The project in which the resource belongs. If it is not provided, the provider project is used.
property region
region?: pulumi.Input<string>;property serviceAccountEmail
serviceAccountEmail?: pulumi.Input<string>;The Service Account email used to create the job.
property subnetwork
subnetwork?: pulumi.Input<string>;The subnetwork to which VMs will be assigned. Should be of the form “regions/REGION/subnetworks/SUBNETWORK”.
property tempGcsLocation
tempGcsLocation: pulumi.Input<string>;A writeable location on GCS for the Dataflow job to dump its temporary data.
property templateGcsPath
templateGcsPath: pulumi.Input<string>;The GCS path to the Dataflow job template.
property zone
zone?: pulumi.Input<string>;The zone in which the created job should run. If it is not provided, the provider zone is used.
interface JobState
interface JobStateInput properties used for looking up and filtering Job resources.
property additionalExperiments
additionalExperiments?: pulumi.Input<pulumi.Input<string>[]>;List of experiments that should be used by the job. An example value is ["enableStackdriverAgentMetrics"].
property ipConfiguration
ipConfiguration?: pulumi.Input<string>;The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE".
property jobId
jobId?: pulumi.Input<string>;The unique ID of this job.
property labels
labels?: pulumi.Input<{[key: string]: any}>;User labels to be specified for the job. Keys and values should follow the restrictions
specified in the labeling restrictions page.
NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided.
Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.
property machineType
machineType?: pulumi.Input<string>;The machine type to use for the job.
property maxWorkers
maxWorkers?: pulumi.Input<number>;The number of workers permitted to work on the job. More workers may improve processing speed at additional cost.
property name
name?: pulumi.Input<string>;A unique name for the resource, required by Dataflow.
property network
network?: pulumi.Input<string>;The network to which VMs will be assigned. If it is not provided, “default” will be used.
property onDelete
onDelete?: pulumi.Input<string>;One of “drain” or “cancel”. Specifies behavior of deletion during pulumi destroy. See above note.
property parameters
parameters?: pulumi.Input<{[key: string]: any}>;Key/Value pairs to be passed to the Dataflow job (as used in the template).
property project
project?: pulumi.Input<string>;The project in which the resource belongs. If it is not provided, the provider project is used.
property region
region?: pulumi.Input<string>;property serviceAccountEmail
serviceAccountEmail?: pulumi.Input<string>;The Service Account email used to create the job.
property state
state?: pulumi.Input<string>;The current state of the resource, selected from the JobState enum
property subnetwork
subnetwork?: pulumi.Input<string>;The subnetwork to which VMs will be assigned. Should be of the form “regions/REGION/subnetworks/SUBNETWORK”.
property tempGcsLocation
tempGcsLocation?: pulumi.Input<string>;A writeable location on GCS for the Dataflow job to dump its temporary data.
property templateGcsPath
templateGcsPath?: pulumi.Input<string>;The GCS path to the Dataflow job template.
property type
type?: pulumi.Input<string>;The type of this job, selected from the JobType enum
property zone
zone?: pulumi.Input<string>;The zone in which the created job should run. If it is not provided, the provider zone is used.