CONDOR_SUBMIT_DAG(1)

NAME
condor_submit_dag - HTCondor Manual

Place a node scheduler job with the local AP to manage job workflow described as a Direct Acyclic Graph (DAG).

SYNOPSIS

condor_submit_dag [-help | -version]

condor_submit_dag [OPTIONS] DAG_file [DAG_file...]

condor_submit_dag [-no_submit] [-v/-verbose] [-f/-force] [-dagman path] [-notification value] [-debug level] [-MaxIdle N] [-MaxJobs N] [-MaxPre N] [-MaxPost N] [-MaxHold N] [-suppress_notification | -dont_suppress_notification] [-SubmitMethod value] [-UseDagDir] [-outfile_dir path] [-config filename] [-Lockfile filename] [-insert_sub_file filename] [-a/-append command] [-batch-name name] [-AutoRescue <0|1>] [-DoRescueFrom N] [-load_save filename] [-DoRecovery] [-AllowVersionMismatch] [-no_recurse | -do_recurse] [-update_submit] [-import_env] [-include_env variable[,variable...]] [-insert_env key=value[;key=value...]] [-valgrind] [-DontAlwaysRunPost | -AlwaysRunPost] [-priority N] [-r/-remote schedd_name] [-schedd-daemon-ad-file filename] [-schedd-address-file filename] [-RescueFile filename]

DESCRIPTION

Automatically produce and place a scheduler universe job to execute HTCondor's DAGMan workflow manager on a list of specified Directed Acyclic Graphs (DAGs) of HTCondor jobs. Extensive documentation is in the HTCondor DAGMan Workflows Section.

OPTIONS

-help
    Display usage information.

-version
    Display version information.

-no_submit
    Produce the HTCondor submit description file for DAGMan, but do not submit DAGMan as an HTCondor job.

-v/-verbose
    Cause condor_submit_dag to give verbose error messages.

-f/-force
    Don't fail to place DAGMan job to AP if previous execution files are discovered. Previous execution files (except the *.dagman.out) are cleaned up and old rescue files are renamed.

-dagman path
    Specify a path to a condor_dagman executable to be executed rather than the installed one discovered in user's path.

-MaxIdle N
    Set the maximum number of idle jobs allowed before placing more jobs to the AP. If not specified then this is set to the value of DAGMAN_MAX_JOBS_IDLE.

    NOTE: condor_dagman can place jobs beyond this threshold in a single job placement cycle, but won't place more jobs until the detected number of idle jobs drops below the specified threshold again.

-MaxJobs N
    Set the maximum number of nodes that have placed a list of jobs to the AP at any given moment. If not specified then this is set to the value of DAGMAN_MAX_JOBS_SUBMITTED.

-MaxPre N
    Set the maximum number of PRE scripts being executed at any given moment. If not specified then this value is set to DAGMAN_MAX_PRE_SCRIPTS.

-MaxPost N
    Set the maximum number of POST scripts being executed at any given moment. If not specified then this value is set to DAGMAN_MAX_POST_SCRIPTS.

-MaxHold N
    Set the maximum number of HOLD scripts being executed at any given moment. If not specified then this value is set to DAGMAN_MAX_HOLD_SCRIPTS.

-notification value
    Set the e-mail notification for DAGMan itself.

-r/-remote schedd_name
    Specify a condor_schedd on a remote machine to place the DAGMan job.

    NOTE: Since DAGMan expects all necessary files to be present in it's working directory, all files used by DAGMan need to be present on the remote machine via some method like a shared filesystem.

-debug level
    Set the DAGMan debug log level (see below for levels).

-UseDagDir
    Inform DAGMan to execute each specified DAG from their respective directories.

-outfile_dir path
    Specify the path to a directory for DAGMan's *.dagman.out debug log to be written.

-config filename
    Specify an HTCondor configuration file to a specific DAGMan instance's execution.

    NOTE: Only one configuration file can be specified which will cause a failure if used in conjunction with the CONFIG command.

-Lockfile filename
    Path to a file to write DAGMan's process information. This prevents other DAGMan processes executing the same DAG(s) from being executed in the same directory.

-insert_sub_file filename
    Specify a file containing JDL submit commands to be inserted in the produced DAGMan job submit description (*.condor.sub).

    NOTE: Only one extra submit description file can be specified. So, the specified file will override any file specified via DAGMAN_INSERT_SUB_FILE

-a/-append command
    Specify JDL commands to insert into the produced DAGMan job submit description (*.condor.sub). Can be used multiple times to add multiple JDL commands. Commands with spaces in them must be enclosed in double quotes.

    NOTE: JDL commands specified via -append take precedence over any commands added via -insert_sub_file or DAGMAN_INSERT_SUB_FILE.

-batch-name name
    Set the JobBatchName to name for DAGMan and all jobs managed by DAGMan.

-AutoRescue <0|1>
    Automatically detect rescue DAG files upon startup and rescue from the most recent (highest number) rescue file discovered. 0 is False. Default 1 is True.

-DoRescueFrom N
    Specify a specific rescue number to locate and restore state from.

-RescueFile filename
    Specify a specific rescue file by name to restore state from.

-load_save filename
    Specify a specific SAVE_POINT_FILE to restore state from. If provided a path DAGMan will attempt to read that file following that path. Otherwise, DAGMan will check for the file in the DAG's save_files sub-directory.

-DoRecovery
    Inform DAGMan to startup in recovery mode (restore state from *.nodes.log).

-AllowVersionMismatch
    Allow a version difference between condor_dagman itself and the *.condor.sub file produced by condor_submit_dag.

    WARNING: This option should only be used if absolutely necessary because version mismatches can cause subtle problems when running DAGMan.

-no_recurse
    Don't recursively pre-produce SUBDAG's submit description files prior to placing root DAG to the AP.

-do_recurse
    Recursively pre-produce all SUBDAG's submit description files prior to placing the root DAG to the AP.

-update_submit
    Don't treat an existing DAGMan submit description file (*.condor.sub) as an error; rather update the file while preserving the -maxjobs, -maxidle, -maxpre, and -maxpost options (if specified).

-import_env
    Inform condor_submit_dag to import the current shell environment into the produced DAGMan submit description file's environment command.

-include_env variable[,variable...]
    Specify a comma separated list of environment variables to add to the produced DAGMan submit description file's getenv command.

-insert_env key=value[;key=value...]
    Specify a delimited string of key=value pairs to explicitly set into the produced DAGMan submit description file's environment command. If the same key is specified multiple times then the last occurrences value takes precedence.

    NOTE: The base delimiter is a semicolon that can be overridden by setting the first character in the string to a valid delimiting character.

    $ condor_submit_dag -insert_env |foo=0|bar=1|baz=2

-valgrind
    Run DAGMan under valgrind.

    NOTE: This is option is intended for testing and development of DAGMan. The DAGMan execution speed is drastically reduced.

    WARNING: Failure will occur if necessary valgrind is not installed. valgrind is only available of Linux OS.

-DontAlwaysRunPost
    Always execute POST scripts even upon failure of any PRE scripts.

-AlwaysRunPost
    Only execute POST scripts after a node's list of jobs have completed (Success or Failure). Default.

-priority N
    Set the minimum JobPrio for jobs managed by DAGMan.

-schedd-daemon-ad-file filename
    Specifies a full path to a daemon ad file for a specific condor_schedd to place the DAGMan job.

-schedd-address-file filename
    Specifies a full path to an address file for a specific condor_schedd to place the DAGMan job.

-suppress_notification
    Suppress email notifications for all jobs managed by DAGMan.

-dont_suppress_notification
    Allow email notifications for any jobs managed by DAGMan that have notifications specified. Default.

-SubmitMethod value
    Specify how DAGMan will place managed jobs to the AP (see values below).

GENERAL REMARKS

NOTE: All command line flags are case insensitive.

Some of the command line options also have corresponding configuration values. The values specified via the command line will take precedence over any configured values.

Some of the command line options are passed down to DAGMan to use when executing SUBDAGs.

  Debug Level Values
level | Details
0 | Never produce output except for usage info.
1 | Very quiet. Only output severe errors.
2 | Normal output and error messages
3 | (Default) Print warnings.
4 | Internal debugging output.
5 | Outer loop debugging.
6 | Inner loop debugging.
7 | Output parsed DAG input lines.

  Submit Method Values
Value | Method
0 | Run condor_submit
1 | Direct place job(s) to local condor_schedd

EXIT STATUS

0 - Success

1 - Failure

EXAMPLES

Execute a single DAG:

$ condor_submit_dag sample.dag

Execute a single DAG that has successfully completed once already:

$ condor_submit_dag -force sample.dag

Execute a DAG with a max threshold of 10 idle jobs:

$ condor_submit_dag -maxidle 10 sample.dag

Execute a DAG with a max limit of 10 PRE scripts and 5 POST scripts executing concurrently:

$ condor_submit_dag -maxpre 10 -maxpost 5 sample.dag

Execute multiple DAGs:

$ condor_submit_dag first.dag second.dag

Execute multiple DAGs in their respective directories:

$ condor_submit_dag -usedagdir subdir1/first.dag subdir2/second.dag

Execute DAG and notify user once DAG is complete:

$ condor_submit_dag -notification complete sample.dag

Execute DAG with a custom batch name:

$ condor_submit_dag -batch-name my-awesome-dag sample.dag

Execute DAG and restore state from specific rescue file:

$ condor_submit_dag -RescueFile simple.dag.rescue008 simple.dag

Execute DAG and restore state from specific rescue number 8:

$ condor_submit_dag -dorescuefrom 8 sample.dag

Execute DAG and restore state from save file post-analysis.save:

$ condor_submit_dag -load_save post-analysis.save sample.dag

Execute DAG and suppress all job e-mail notifications:

$ condor_submit_dag -suppress_notification sample.dag

SEE ALSO

None

AVAILABILITY

Linux, MacOS, Windows

AUTHOR
HTCondor Team

COPYRIGHT
1990-2026, Center for High Throughput Computing, Computer Sciences Department, University of Wisconsin-Madison, Madison, WI, US. Licensed under the Apache License, Version 2.0.