Class Job

Creates a job on Dataflow, which is an implementation of Apache Beam running on Google Compute Engine. For more information see the official documentation for Beam and Dataflow.

Example Usage

using Pulumi;
using Gcp = Pulumi.Gcp;

class MyStack : Stack
{
public MyStack()
{
    var bigDataJob = new Gcp.Dataflow.Job(&quot;bigDataJob&quot;, new Gcp.Dataflow.JobArgs
    {
        Parameters = 
        {
            { &quot;baz&quot;, &quot;qux&quot; },
            { &quot;foo&quot;, &quot;bar&quot; },
        },
        TempGcsLocation = &quot;gs://my-bucket/tmp_dir&quot;,
        TemplateGcsPath = &quot;gs://my-bucket/templates/template_file&quot;,
    });
}

}

Note on "destroy" / "apply"

There are many types of Dataflow jobs. Some Dataflow jobs run constantly, getting new data from (e.g.) a GCS bucket, and outputting data continuously. Some jobs process a set amount of data then terminate. All jobs can fail while running due to programming errors or other issues. In this way, Dataflow jobs are different from most other Google resources.

The Dataflow resource is considered 'existing' while it is in a nonterminal state. If it reaches a terminal state (e.g. 'FAILED', 'COMPLETE', 'CANCELLED'), it will be recreated on the next 'apply'. This is as expected for jobs which run continuously, but may surprise users who use this resource for other kinds of Dataflow jobs.

A Dataflow job which is 'destroyed' may be "cancelled" or "drained". If "cancelled", the job terminates - any data written remains where it is, but no new data will be processed. If "drained", no new data will enter the pipeline, but any data currently in the pipeline will finish being processed. The default is "cancelled", but if a user sets on_delete to "drain" in the configuration, you may experience a long wait for your pulumi destroy to complete.

Inheritance

System.Object

Resource

CustomResource

Job

Inherited Members

CustomResource.Id

Resource.GetResourceType()

Resource.GetResourceName()

Resource.Urn

System.Object.Equals(System.Object)

System.Object.Equals(System.Object, System.Object)

System.Object.GetHashCode()

System.Object.GetType()

System.Object.MemberwiseClone()

System.Object.ReferenceEquals(System.Object, System.Object)

System.Object.ToString()

Namespace: Pulumi.Gcp.Dataflow

Assembly: Pulumi.Gcp.dll

Syntax

public class Job : CustomResource

Constructors

View Source

Job(String, JobArgs, CustomResourceOptions)

Create a Job resource with the given unique name, arguments, and options.

Declaration

public Job(string name, JobArgs args, CustomResourceOptions options = null)

Parameters

Type	Name	Description
System.String	name	The unique name of the resource
JobArgs	args	The arguments used to populate this resource's properties
CustomResourceOptions	options	A bag of options that control this resource's behavior

Properties

View Source

AdditionalExperiments

List of experiments that should be used by the job. An example value is ["enable_stackdriver_agent_metrics"].

Declaration

public Output<ImmutableArray<string>> AdditionalExperiments { get; }

Property Value

Type	Description
Output<System.Collections.Immutable.ImmutableArray<System.String>>

View Source

IpConfiguration

The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE".

Declaration

public Output<string> IpConfiguration { get; }

Property Value

Type	Description
Output<System.String>

View Source

JobId

The unique ID of this job.

Declaration

public Output<string> JobId { get; }

Property Value

Type	Description
Output<System.String>

View Source

Labels

User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.

Declaration

public Output<ImmutableDictionary<string, object>> Labels { get; }

Property Value

Type	Description
Output<System.Collections.Immutable.ImmutableDictionary<System.String, System.Object>>

View Source

MachineType

The machine type to use for the job.

Declaration

public Output<string> MachineType { get; }

Property Value

Type	Description
Output<System.String>

View Source

MaxWorkers

The number of workers permitted to work on the job. More workers may improve processing speed at additional cost.

Declaration

public Output<int?> MaxWorkers { get; }

Property Value

Type	Description
Output<System.Nullable<System.Int32>>

View Source

Name

A unique name for the resource, required by Dataflow.

Declaration

public Output<string> Name { get; }

Property Value

Type	Description
Output<System.String>

View Source

Network

The network to which VMs will be assigned. If it is not provided, "default" will be used.

Declaration

public Output<string> Network { get; }

Property Value

Type	Description
Output<System.String>

View Source

OnDelete

One of "drain" or "cancel". Specifies behavior of deletion during pulumi destroy. See above note.

Declaration

public Output<string> OnDelete { get; }

Property Value

Type	Description
Output<System.String>

View Source

Parameters

Key/Value pairs to be passed to the Dataflow job (as used in the template).

Declaration

public Output<ImmutableDictionary<string, object>> Parameters { get; }

Property Value

Type	Description
Output<System.Collections.Immutable.ImmutableDictionary<System.String, System.Object>>

View Source

Project

The project in which the resource belongs. If it is not provided, the provider project is used.

Declaration

public Output<string> Project { get; }

Property Value

Type	Description
Output<System.String>

View Source

Region

Declaration

public Output<string> Region { get; }

Property Value

Type	Description
Output<System.String>

View Source

ServiceAccountEmail

The Service Account email used to create the job.

Declaration

public Output<string> ServiceAccountEmail { get; }

Property Value

Type	Description
Output<System.String>

View Source

State

The current state of the resource, selected from the JobState enum

Declaration

public Output<string> State { get; }

Property Value

Type	Description
Output<System.String>

View Source

Subnetwork

The subnetwork to which VMs will be assigned. Should be of the form "regions/REGION/subnetworks/SUBNETWORK".

Declaration

public Output<string> Subnetwork { get; }

Property Value

Type	Description
Output<System.String>

View Source

TempGcsLocation

A writeable location on GCS for the Dataflow job to dump its temporary data.

Declaration

public Output<string> TempGcsLocation { get; }

Property Value

Type	Description
Output<System.String>

View Source

TemplateGcsPath

The GCS path to the Dataflow job template.

Declaration

public Output<string> TemplateGcsPath { get; }

Property Value

Type	Description
Output<System.String>

View Source

Type

The type of this job, selected from the JobType enum

Declaration

public Output<string> Type { get; }

Property Value

Type	Description
Output<System.String>

View Source

Zone

The zone in which the created job should run. If it is not provided, the provider zone is used.

Declaration

public Output<string> Zone { get; }

Property Value

Type	Description
Output<System.String>

Methods

View Source

Get(String, Input<String>, JobState, CustomResourceOptions)

Get an existing Job resource's state with the given name, ID, and optional extra properties used to qualify the lookup.

Declaration

public static Job Get(string name, Input<string> id, JobState state = null, CustomResourceOptions options = null)

Parameters

Type	Name	Description
System.String	name	The unique name of the resulting resource.
Input<System.String>	id	The unique provider ID of the resource to lookup.
JobState	state	Any extra arguments used during the lookup.
CustomResourceOptions	options	A bag of options that control this resource's behavior

Returns

Type	Description
Job