Job

Manages a job resource within a Dataproc cluster within GCE. For more information see the official dataproc documentation.

!> Note: This resource does not support ‘update’ and changing any attributes will cause the resource to be recreated.

Create a Job Resource

new Job(name: string, args: JobArgs, opts?: CustomResourceOptions);

def Job(resource_name, opts=None, force_delete=None, hadoop_config=None, hive_config=None, labels=None, pig_config=None, placement=None, project=None, pyspark_config=None, reference=None, region=None, scheduling=None, spark_config=None, sparksql_config=None, __props__=None);

func NewJob(ctx *Context, name string, args JobArgs, opts ...ResourceOption) (*Job, error)

public Job(string name, JobArgs args, CustomResourceOptions? opts = null)

name string: The unique name of the resource.
args JobArgs: The arguments to resource properties.
opts CustomResourceOptions: Bag of options to control resource's behavior.

resource_name str: The unique name of the resource.
opts ResourceOptions: A bag of options that control this resource's behavior.

ctx Context: Context object for the current deployment.
name string: The unique name of the resource.
args JobArgs: The arguments to resource properties.
opts ResourceOption: Bag of options to control resource's behavior.

name string: The unique name of the resource.
args JobArgs: The arguments to resource properties.
opts CustomResourceOptions: Bag of options to control resource's behavior.

Job Resource Properties

To learn more about resource properties and how to use them, see Inputs and Outputs in the Programming Model docs.

Inputs

The Job resource accepts the following input properties:

Placement JobPlacementArgs: The config of job placement.
ForceDelete bool: By default, you can only delete inactive jobs within Dataproc. Setting this to true, and calling destroy, will ensure that the job is first cancelled before issuing the delete.
HadoopConfig JobHadoopConfigArgs: The config of Hadoop job
HiveConfig JobHiveConfigArgs: The config of hive job
Labels Dictionary<string, string>: The list of labels (key/value pairs) to add to the job.
PigConfig JobPigConfigArgs: The config of pag job.
Project string: The project in which the cluster can be found and jobs subsequently run against. If it is not provided, the provider project is used.
PysparkConfig JobPysparkConfigArgs: The config of pySpark job.
Reference JobReferenceArgs: The reference of the job
Region string: The Cloud Dataproc region. This essentially determines which clusters are available for this job to be submitted to. If not specified, defaults to global.
Scheduling JobSchedulingArgs: Optional. Job scheduling configuration.
SparkConfig JobSparkConfigArgs: The config of the Spark job.
SparksqlConfig JobSparksqlConfigArgs: The config of SparkSql job

Placement JobPlacement: The config of job placement.
ForceDelete bool: By default, you can only delete inactive jobs within Dataproc. Setting this to true, and calling destroy, will ensure that the job is first cancelled before issuing the delete.
HadoopConfig JobHadoopConfig: The config of Hadoop job
HiveConfig JobHiveConfig: The config of hive job
Labels map[string]string: The list of labels (key/value pairs) to add to the job.
PigConfig JobPigConfig: The config of pag job.
Project string: The project in which the cluster can be found and jobs subsequently run against. If it is not provided, the provider project is used.
PysparkConfig JobPysparkConfig: The config of pySpark job.
Reference JobReference: The reference of the job
Region string: The Cloud Dataproc region. This essentially determines which clusters are available for this job to be submitted to. If not specified, defaults to global.
Scheduling JobScheduling: Optional. Job scheduling configuration.
SparkConfig JobSparkConfig: The config of the Spark job.
SparksqlConfig JobSparksqlConfig: The config of SparkSql job

placement JobPlacement: The config of job placement.
forceDelete boolean: By default, you can only delete inactive jobs within Dataproc. Setting this to true, and calling destroy, will ensure that the job is first cancelled before issuing the delete.
hadoopConfig JobHadoopConfig: The config of Hadoop job
hiveConfig JobHiveConfig: The config of hive job
labels {[key: string]: string}: The list of labels (key/value pairs) to add to the job.
pigConfig JobPigConfig: The config of pag job.
project string: The project in which the cluster can be found and jobs subsequently run against. If it is not provided, the provider project is used.
pysparkConfig JobPysparkConfig: The config of pySpark job.
reference JobReference: The reference of the job
region string: The Cloud Dataproc region. This essentially determines which clusters are available for this job to be submitted to. If not specified, defaults to global.
scheduling JobScheduling: Optional. Job scheduling configuration.
sparkConfig JobSparkConfig: The config of the Spark job.
sparksqlConfig JobSparksqlConfig: The config of SparkSql job

placement Dict[JobPlacement]: The config of job placement.
force_delete bool: By default, you can only delete inactive jobs within Dataproc. Setting this to true, and calling destroy, will ensure that the job is first cancelled before issuing the delete.
hadoop_config Dict[JobHadoopConfig]: The config of Hadoop job
hive_config Dict[JobHiveConfig]: The config of hive job
labels Dict[str, str]: The list of labels (key/value pairs) to add to the job.
pig_config Dict[JobPigConfig]: The config of pag job.
project str: The project in which the cluster can be found and jobs subsequently run against. If it is not provided, the provider project is used.
pyspark_config Dict[JobPysparkConfig]: The config of pySpark job.
reference Dict[JobReference]: The reference of the job
region str: The Cloud Dataproc region. This essentially determines which clusters are available for this job to be submitted to. If not specified, defaults to global.
scheduling Dict[JobScheduling]: Optional. Job scheduling configuration.
spark_config Dict[JobSparkConfig]: The config of the Spark job.
sparksql_config Dict[JobSparksqlConfig]: The config of SparkSql job

Outputs

All input properties are implicitly available as output properties. Additionally, the Job resource produces the following output properties:

DriverControlsFilesUri string: If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.
DriverOutputResourceUri string: A URI pointing to the location of the stdout of the job’s driver program.
Id string: The provider-assigned unique ID for this managed resource.
Status JobStatus: The status of the job.

DriverControlsFilesUri string: If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.
DriverOutputResourceUri string: A URI pointing to the location of the stdout of the job’s driver program.
Id string: The provider-assigned unique ID for this managed resource.
Status JobStatus: The status of the job.

driverControlsFilesUri string: If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.
driverOutputResourceUri string: A URI pointing to the location of the stdout of the job’s driver program.
id string: The provider-assigned unique ID for this managed resource.
status JobStatus: The status of the job.

driver_controls_files_uri str: If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.
driver_output_resource_uri str: A URI pointing to the location of the stdout of the job’s driver program.
id str: The provider-assigned unique ID for this managed resource.
status Dict[JobStatus]: The status of the job.

Look up an Existing Job Resource

Get an existing Job resource’s state with the given name, ID, and optional extra properties used to qualify the lookup.

public static get(name: string, id: Input<ID>, state?: JobState, opts?: CustomResourceOptions): Job

static get(resource_name, id, opts=None, driver_controls_files_uri=None, driver_output_resource_uri=None, force_delete=None, hadoop_config=None, hive_config=None, labels=None, pig_config=None, placement=None, project=None, pyspark_config=None, reference=None, region=None, scheduling=None, spark_config=None, sparksql_config=None, status=None, __props__=None);

func GetJob(ctx *Context, name string, id IDInput, state *JobState, opts ...ResourceOption) (*Job, error)

public static Job Get(string name, Input<string> id, JobState? state, CustomResourceOptions? opts = null)

name: The unique name of the resulting resource.
id: The unique provider ID of the resource to lookup.
state: Any extra arguments used during the lookup.
opts: A bag of options that control this resource's behavior.

resource_name: The unique name of the resulting resource.
id: The unique provider ID of the resource to lookup.

name: The unique name of the resulting resource.
id: The unique provider ID of the resource to lookup.
state: Any extra arguments used during the lookup.
opts: A bag of options that control this resource's behavior.

name: The unique name of the resulting resource.
id: The unique provider ID of the resource to lookup.
state: Any extra arguments used during the lookup.
opts: A bag of options that control this resource's behavior.

The following state arguments are supported:

DriverControlsFilesUri string: If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.
DriverOutputResourceUri string: A URI pointing to the location of the stdout of the job’s driver program.
ForceDelete bool: By default, you can only delete inactive jobs within Dataproc. Setting this to true, and calling destroy, will ensure that the job is first cancelled before issuing the delete.
HadoopConfig JobHadoopConfigArgs: The config of Hadoop job
HiveConfig JobHiveConfigArgs: The config of hive job
Labels Dictionary<string, string>: The list of labels (key/value pairs) to add to the job.
PigConfig JobPigConfigArgs: The config of pag job.
Placement JobPlacementArgs: The config of job placement.
Project string: The project in which the cluster can be found and jobs subsequently run against. If it is not provided, the provider project is used.
PysparkConfig JobPysparkConfigArgs: The config of pySpark job.
Reference JobReferenceArgs: The reference of the job
Region string: The Cloud Dataproc region. This essentially determines which clusters are available for this job to be submitted to. If not specified, defaults to global.
Scheduling JobSchedulingArgs: Optional. Job scheduling configuration.
SparkConfig JobSparkConfigArgs: The config of the Spark job.
SparksqlConfig JobSparksqlConfigArgs: The config of SparkSql job
Status JobStatusArgs: The status of the job.

DriverControlsFilesUri string: If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.
DriverOutputResourceUri string: A URI pointing to the location of the stdout of the job’s driver program.
ForceDelete bool: By default, you can only delete inactive jobs within Dataproc. Setting this to true, and calling destroy, will ensure that the job is first cancelled before issuing the delete.
HadoopConfig JobHadoopConfig: The config of Hadoop job
HiveConfig JobHiveConfig: The config of hive job
Labels map[string]string: The list of labels (key/value pairs) to add to the job.
PigConfig JobPigConfig: The config of pag job.
Placement JobPlacement: The config of job placement.
Project string: The project in which the cluster can be found and jobs subsequently run against. If it is not provided, the provider project is used.
PysparkConfig JobPysparkConfig: The config of pySpark job.
Reference JobReference: The reference of the job
Region string: The Cloud Dataproc region. This essentially determines which clusters are available for this job to be submitted to. If not specified, defaults to global.
Scheduling JobScheduling: Optional. Job scheduling configuration.
SparkConfig JobSparkConfig: The config of the Spark job.
SparksqlConfig JobSparksqlConfig: The config of SparkSql job
Status JobStatus: The status of the job.

driverControlsFilesUri string: If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.
driverOutputResourceUri string: A URI pointing to the location of the stdout of the job’s driver program.
forceDelete boolean: By default, you can only delete inactive jobs within Dataproc. Setting this to true, and calling destroy, will ensure that the job is first cancelled before issuing the delete.
hadoopConfig JobHadoopConfig: The config of Hadoop job
hiveConfig JobHiveConfig: The config of hive job
labels {[key: string]: string}: The list of labels (key/value pairs) to add to the job.
pigConfig JobPigConfig: The config of pag job.
placement JobPlacement: The config of job placement.
project string: The project in which the cluster can be found and jobs subsequently run against. If it is not provided, the provider project is used.
pysparkConfig JobPysparkConfig: The config of pySpark job.
reference JobReference: The reference of the job
region string: The Cloud Dataproc region. This essentially determines which clusters are available for this job to be submitted to. If not specified, defaults to global.
scheduling JobScheduling: Optional. Job scheduling configuration.
sparkConfig JobSparkConfig: The config of the Spark job.
sparksqlConfig JobSparksqlConfig: The config of SparkSql job
status JobStatus: The status of the job.

driver_controls_files_uri str: If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.
driver_output_resource_uri str: A URI pointing to the location of the stdout of the job’s driver program.
force_delete bool: By default, you can only delete inactive jobs within Dataproc. Setting this to true, and calling destroy, will ensure that the job is first cancelled before issuing the delete.
hadoop_config Dict[JobHadoopConfig]: The config of Hadoop job
hive_config Dict[JobHiveConfig]: The config of hive job
labels Dict[str, str]: The list of labels (key/value pairs) to add to the job.
pig_config Dict[JobPigConfig]: The config of pag job.
placement Dict[JobPlacement]: The config of job placement.
project str: The project in which the cluster can be found and jobs subsequently run against. If it is not provided, the provider project is used.
pyspark_config Dict[JobPysparkConfig]: The config of pySpark job.
reference Dict[JobReference]: The reference of the job
region str: The Cloud Dataproc region. This essentially determines which clusters are available for this job to be submitted to. If not specified, defaults to global.
scheduling Dict[JobScheduling]: Optional. Job scheduling configuration.
spark_config Dict[JobSparkConfig]: The config of the Spark job.
sparksql_config Dict[JobSparksqlConfig]: The config of SparkSql job
status Dict[JobStatus]: The status of the job.

Supporting Types

JobHadoopConfig

ArchiveUris List<string>: HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
Args List<string>: The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
FileUris List<string>: HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
JarFileUris List<string>: HCFS URIs of jar files to be added to the Spark CLASSPATH.
LoggingConfig JobHadoopConfigLoggingConfigArgs
MainClass string: The name of the driver’s main class. The jar file containing the class must be in the default CLASSPATH or specified in jar_file_uris. Conflicts with main_jar_file_uri
MainJarFileUri string: The HCFS URI of the jar file containing the main class. Examples: ‘gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar’ ‘hdfs:/tmp/test-samples/custom-wordcount.jar’ ‘file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar’. Conflicts with main_class
Properties Dictionary<string, string>: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.

ArchiveUris []string: HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
Args []string: The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
FileUris []string: HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
JarFileUris []string: HCFS URIs of jar files to be added to the Spark CLASSPATH.
LoggingConfig JobHadoopConfigLoggingConfig
MainClass string: The name of the driver’s main class. The jar file containing the class must be in the default CLASSPATH or specified in jar_file_uris. Conflicts with main_jar_file_uri
MainJarFileUri string: The HCFS URI of the jar file containing the main class. Examples: ‘gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar’ ‘hdfs:/tmp/test-samples/custom-wordcount.jar’ ‘file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar’. Conflicts with main_class
Properties map[string]string: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.

archiveUris string[]: HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
args string[]: The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
fileUris string[]: HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
jarFileUris string[]: HCFS URIs of jar files to be added to the Spark CLASSPATH.
loggingConfig JobHadoopConfigLoggingConfig
mainClass string: The name of the driver’s main class. The jar file containing the class must be in the default CLASSPATH or specified in jar_file_uris. Conflicts with main_jar_file_uri
mainJarFileUri string: The HCFS URI of the jar file containing the main class. Examples: ‘gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar’ ‘hdfs:/tmp/test-samples/custom-wordcount.jar’ ‘file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar’. Conflicts with main_class
properties {[key: string]: string}: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.

archiveUris List[str]: HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
args List[str]: The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
fileUris List[str]: HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
jarFileUris List[str]: HCFS URIs of jar files to be added to the Spark CLASSPATH.
loggingConfig Dict[JobHadoopConfigLoggingConfig]
mainClass str: The name of the driver’s main class. The jar file containing the class must be in the default CLASSPATH or specified in jar_file_uris. Conflicts with main_jar_file_uri
mainJarFileUri str: The HCFS URI of the jar file containing the main class. Examples: ‘gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar’ ‘hdfs:/tmp/test-samples/custom-wordcount.jar’ ‘file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar’. Conflicts with main_class
properties Dict[str, str]: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.

JobHadoopConfigLoggingConfig

DriverLogLevels Dictionary<string, string>

DriverLogLevels map[string]string

driverLogLevels {[key: string]: string}

driverLogLevels Dict[str, str]

JobHiveConfig

ContinueOnFailure bool: Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries. Defaults to false.
JarFileUris List<string>: HCFS URIs of jar files to be added to the Spark CLASSPATH.
Properties Dictionary<string, string>: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
QueryFileUri string: The HCFS URI of the script that contains SQL queries. Conflicts with query_list
QueryLists List<string>: The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri
ScriptVariables Dictionary<string, string>: Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).

ContinueOnFailure bool: Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries. Defaults to false.
JarFileUris []string: HCFS URIs of jar files to be added to the Spark CLASSPATH.
Properties map[string]string: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
QueryFileUri string: The HCFS URI of the script that contains SQL queries. Conflicts with query_list
QueryLists []string: The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri
ScriptVariables map[string]string: Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).

continueOnFailure boolean: Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries. Defaults to false.
jarFileUris string[]: HCFS URIs of jar files to be added to the Spark CLASSPATH.
properties {[key: string]: string}: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
queryFileUri string: The HCFS URI of the script that contains SQL queries. Conflicts with query_list
queryLists string[]: The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri
scriptVariables {[key: string]: string}: Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).

continueOnFailure bool: Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries. Defaults to false.
jarFileUris List[str]: HCFS URIs of jar files to be added to the Spark CLASSPATH.
properties Dict[str, str]: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
queryFileUri str: The HCFS URI of the script that contains SQL queries. Conflicts with query_list
queryLists List[str]: The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri
scriptVariables Dict[str, str]: Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).

JobPigConfig

ContinueOnFailure bool: Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries. Defaults to false.
JarFileUris List<string>: HCFS URIs of jar files to be added to the Spark CLASSPATH.
LoggingConfig JobPigConfigLoggingConfigArgs
Properties Dictionary<string, string>: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
QueryFileUri string: The HCFS URI of the script that contains SQL queries. Conflicts with query_list
QueryLists List<string>: The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri
ScriptVariables Dictionary<string, string>: Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).

ContinueOnFailure bool: Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries. Defaults to false.
JarFileUris []string: HCFS URIs of jar files to be added to the Spark CLASSPATH.
LoggingConfig JobPigConfigLoggingConfig
Properties map[string]string: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
QueryFileUri string: The HCFS URI of the script that contains SQL queries. Conflicts with query_list
QueryLists []string: The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri
ScriptVariables map[string]string: Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).

continueOnFailure boolean: Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries. Defaults to false.
jarFileUris string[]: HCFS URIs of jar files to be added to the Spark CLASSPATH.
loggingConfig JobPigConfigLoggingConfig
properties {[key: string]: string}: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
queryFileUri string: The HCFS URI of the script that contains SQL queries. Conflicts with query_list
queryLists string[]: The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri
scriptVariables {[key: string]: string}: Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).

continueOnFailure bool: Whether to continue executing queries if a query fails. The default value is false. Setting to true can be useful when executing independent parallel queries. Defaults to false.
jarFileUris List[str]: HCFS URIs of jar files to be added to the Spark CLASSPATH.
loggingConfig Dict[JobPigConfigLoggingConfig]
properties Dict[str, str]: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
queryFileUri str: The HCFS URI of the script that contains SQL queries. Conflicts with query_list
queryLists List[str]: The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri
scriptVariables Dict[str, str]: Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).

JobPigConfigLoggingConfig

DriverLogLevels Dictionary<string, string>

DriverLogLevels map[string]string

driverLogLevels {[key: string]: string}

driverLogLevels Dict[str, str]

JobPlacement

ClusterName string
ClusterUuid string

ClusterName string
ClusterUuid string

clusterName string
clusterUuid string

clusterName str
clusterUuid str

JobPysparkConfig

MainPythonFileUri string: The HCFS URI of the main Python file to use as the driver. Must be a .py file.
ArchiveUris List<string>: HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
Args List<string>: The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
FileUris List<string>: HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
JarFileUris List<string>: HCFS URIs of jar files to be added to the Spark CLASSPATH.
LoggingConfig JobPysparkConfigLoggingConfigArgs
Properties Dictionary<string, string>: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
PythonFileUris List<string>: HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.

MainPythonFileUri string: The HCFS URI of the main Python file to use as the driver. Must be a .py file.
ArchiveUris []string: HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
Args []string: The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
FileUris []string: HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
JarFileUris []string: HCFS URIs of jar files to be added to the Spark CLASSPATH.
LoggingConfig JobPysparkConfigLoggingConfig
Properties map[string]string: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
PythonFileUris []string: HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.

mainPythonFileUri string: The HCFS URI of the main Python file to use as the driver. Must be a .py file.
archiveUris string[]: HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
args string[]: The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
fileUris string[]: HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
jarFileUris string[]: HCFS URIs of jar files to be added to the Spark CLASSPATH.
loggingConfig JobPysparkConfigLoggingConfig
properties {[key: string]: string}: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
pythonFileUris string[]: HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.

mainPythonFileUri str: The HCFS URI of the main Python file to use as the driver. Must be a .py file.
archiveUris List[str]: HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
args List[str]: The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
fileUris List[str]: HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
jarFileUris List[str]: HCFS URIs of jar files to be added to the Spark CLASSPATH.
loggingConfig Dict[JobPysparkConfigLoggingConfig]
properties Dict[str, str]: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
pythonFileUris List[str]: HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.

JobPysparkConfigLoggingConfig

DriverLogLevels Dictionary<string, string>

DriverLogLevels map[string]string

driverLogLevels {[key: string]: string}

driverLogLevels Dict[str, str]

JobReference

JobId string

JobId string

jobId string

job_id str

JobScheduling

MaxFailuresPerHour int

MaxFailuresPerHour int

maxFailuresPerHour number

maxFailuresPerHour float

JobSparkConfig

ArchiveUris List<string>: HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
Args List<string>: The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
FileUris List<string>: HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
JarFileUris List<string>: HCFS URIs of jar files to be added to the Spark CLASSPATH.
LoggingConfig JobSparkConfigLoggingConfigArgs
MainClass string: The name of the driver’s main class. The jar file containing the class must be in the default CLASSPATH or specified in jar_file_uris. Conflicts with main_jar_file_uri
MainJarFileUri string: The HCFS URI of the jar file containing the main class. Examples: ‘gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar’ ‘hdfs:/tmp/test-samples/custom-wordcount.jar’ ‘file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar’. Conflicts with main_class
Properties Dictionary<string, string>: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.

ArchiveUris []string: HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
Args []string: The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
FileUris []string: HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
JarFileUris []string: HCFS URIs of jar files to be added to the Spark CLASSPATH.
LoggingConfig JobSparkConfigLoggingConfig
MainClass string: The name of the driver’s main class. The jar file containing the class must be in the default CLASSPATH or specified in jar_file_uris. Conflicts with main_jar_file_uri
MainJarFileUri string: The HCFS URI of the jar file containing the main class. Examples: ‘gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar’ ‘hdfs:/tmp/test-samples/custom-wordcount.jar’ ‘file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar’. Conflicts with main_class
Properties map[string]string: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.

archiveUris string[]: HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
args string[]: The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
fileUris string[]: HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
jarFileUris string[]: HCFS URIs of jar files to be added to the Spark CLASSPATH.
loggingConfig JobSparkConfigLoggingConfig
mainClass string: The name of the driver’s main class. The jar file containing the class must be in the default CLASSPATH or specified in jar_file_uris. Conflicts with main_jar_file_uri
mainJarFileUri string: The HCFS URI of the jar file containing the main class. Examples: ‘gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar’ ‘hdfs:/tmp/test-samples/custom-wordcount.jar’ ‘file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar’. Conflicts with main_class
properties {[key: string]: string}: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.

archiveUris List[str]: HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
args List[str]: The arguments to pass to the driver. Do not include arguments, such as -libjars or -Dfoo=bar, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
fileUris List[str]: HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
jarFileUris List[str]: HCFS URIs of jar files to be added to the Spark CLASSPATH.
loggingConfig Dict[JobSparkConfigLoggingConfig]
mainClass str: The name of the driver’s main class. The jar file containing the class must be in the default CLASSPATH or specified in jar_file_uris. Conflicts with main_jar_file_uri
mainJarFileUri str: The HCFS URI of the jar file containing the main class. Examples: ‘gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar’ ‘hdfs:/tmp/test-samples/custom-wordcount.jar’ ‘file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar’. Conflicts with main_class
properties Dict[str, str]: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.

JobSparkConfigLoggingConfig

DriverLogLevels Dictionary<string, string>

DriverLogLevels map[string]string

driverLogLevels {[key: string]: string}

driverLogLevels Dict[str, str]

JobSparksqlConfig

JarFileUris List<string>: HCFS URIs of jar files to be added to the Spark CLASSPATH.
LoggingConfig JobSparksqlConfigLoggingConfigArgs
Properties Dictionary<string, string>: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
QueryFileUri string: The HCFS URI of the script that contains SQL queries. Conflicts with query_list
QueryLists List<string>: The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri
ScriptVariables Dictionary<string, string>: Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).

JarFileUris []string: HCFS URIs of jar files to be added to the Spark CLASSPATH.
LoggingConfig JobSparksqlConfigLoggingConfig
Properties map[string]string: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
QueryFileUri string: The HCFS URI of the script that contains SQL queries. Conflicts with query_list
QueryLists []string: The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri
ScriptVariables map[string]string: Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).

jarFileUris string[]: HCFS URIs of jar files to be added to the Spark CLASSPATH.
loggingConfig JobSparksqlConfigLoggingConfig
properties {[key: string]: string}: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
queryFileUri string: The HCFS URI of the script that contains SQL queries. Conflicts with query_list
queryLists string[]: The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri
scriptVariables {[key: string]: string}: Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).

jarFileUris List[str]: HCFS URIs of jar files to be added to the Spark CLASSPATH.
loggingConfig Dict[JobSparksqlConfigLoggingConfig]
properties Dict[str, str]: A mapping of property names to values, used to configure Spark SQL’s SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
queryFileUri str: The HCFS URI of the script that contains SQL queries. Conflicts with query_list
queryLists List[str]: The list of SQL queries or statements to execute as part of the job. Conflicts with query_file_uri
scriptVariables Dict[str, str]: Mapping of query variable names to values (equivalent to the Spark SQL command: SET name="value";).

JobSparksqlConfigLoggingConfig

DriverLogLevels Dictionary<string, string>

DriverLogLevels map[string]string

driverLogLevels {[key: string]: string}

driverLogLevels Dict[str, str]

JobStatus

Details string
State string
StateStartTime string
Substate string

Details string
State string
StateStartTime string
Substate string

details string
state string
stateStartTime string
substate string

details str
state str
stateStartTime str
substate str

Package Details

Repository: https://github.com/pulumi/pulumi-gcp
License: Apache-2.0
Notes: This Pulumi package is based on the google-beta Terraform Provider.

The Pulumi Platform

Get Started

Migrate to Pulumi

Solutions for All Teams and Engineers

Any Code

Any Cloud

Job