Open MPI logo

prun(1) man page (version 5.0.0rc2)

  |   Home   |   Support   |   FAQ   |  

« Return to documentation listing

Table of Contents

Name

prun - Execute serial and parallel jobs with the PMIx Reference Runtime (PRTE).

Synopsis

C]prunR] requires a running C]prteR] Distributed Virtual Machine (DVM) to be running at the time of the call. See prte(1) for more information.

Single Process Multiple Data (SPMD) Model:


C]prun [ options ] <program> [ <args> ]R]Multiple Instruction Multiple Data (MIMD) Model:


C]prun [ global_options ] [rs] [ local_options1 ] <program1> [ <args1> ] : [rs] [ local_options2 ] <program2> [ <args2> ] : [rs] ... : [rs] [ local_optionsN ] <programN> [ <argsN> ]R]Note that in both models, invoking C]prunR] via an absolute path
name is equivalent to specifying the C]--prefixR] option with a C]<dir>R] value equivalent to the directory where C]prunR] resides, minus its last subdirectory. For example:

C]$ /usr/local/bin/prun ...R]is equivalent to


C]$ prun --prefix /usr/localR]Quick Summary

If you are simply looking for how to run an application,

you probably
want to use a command line of the following form:

C]$ prun [ -np X ] [ --hostfile <filename> ] <program>R]This will run C]XR] copies of C]<program>R] in your current
run-time environment over the set of hosts specified by C]<filename>R], scheduling (by default) in a round-robin fashion by CPU slot. If running under a supported resource manager a hostfile is usually not required unless the caller wishes to further restrict the set of resources used for that job.

Please note that PRTE automatically binds processes. See prte-map(1) for defaults for the mapping, ranking, and binding of processes.

If your application uses threads, then you probably want to ensure that you are either not bound at all (by specifying C]--bind-to noneR]), or bound to multiple cores using an appropriate binding level or specific number of processing elements per application process.

Default ranking is by C]slotR] if number of processes <= 2, otherwise default to ranking by C]packageR] (formally known as [lq]socket[rq]).

See prte-map(1) for more details on mapping, ranking, and binding options.

Options

This section includes many commonly used options. There may be other options listed with C]prun --helpR].

C]prunR] will send the name of the directory where it was invoked on the local node to each of the remote nodes, and attempt to change to that directory. See the [lq]Current Working Directory[rq] section below for further details.

B]CB]<program>B]R]
The program executable. This is identified as the first non-recognized argument to C]prunR].
B]CB]<args>B]R]
Pass these run-time arguments to every new process. These must always be the last arguments to C]prunR] after the C]<program>R]. If an app context file is used, C]<args>R] will be ignored.
B]CB]-h, --helpB]R]
Display help for this command
B]CB]-q, --quietB]R]
Suppress informative messages from C]prunR] during application execution.
B]CB]-v, --verboseB]R]
Be verbose
B]CB]-V, --versionB]R]
Print version number. If no other arguments are given, this will also cause C]prunR] to exit.

Specifying Number of Processes

The following options specify the number of processes to launch. Note that none of the options imply a particular binding policy - e.g., requesting N processes for each socket does not imply that the processes will be bound to the package.

Additional options and details are presented in prte-map(1). Below are a few of the commonly used options.

B]CB]-c, -n, --n, --np <#>B]R]
Run this many copies of the program on the given nodes. This option indicates that the specified file is an executable program and not an application context. If no value is provided for the number of copies to execute (i.e., neither the C]--npR] nor its synonyms are provided on the command line), C]prunR] will automatically execute a copy of the program on each process slot (see below for description of a [lq]process slot[rq]). This feature, however, can only be used in the SPMD model and will return an error (without beginning execution of the application) otherwise.

I/O Management

To manage standard I/O:

B]CB]--output-filename <filename>B]R]
Redirect the C]stdoutR], C]stderrR], and C]stddiagR] of all processes to a process-unique version of the specified filename ([lq]filename.id[rq]). Any directories in the filename will automatically be created. Each output file will consist of [lq]filename.id[rq], where the C]idR] will be the processes[cq] rank, left-filled with zero[cq]s for correct ordering in listings. Both C]stdoutR] and C]stderrR] will be redirected to the file. A relative path value will be converted to an absolute path based on the current working directory where C]prunR] is executed. Note that this I]will notR] work in environments where the file system on compute nodes differs from that where C]prunR] is executed. This option accepts one case-insensitive directive, specified after a colon (C]:R]): C]NOCOPYR] indicates that the output is not to be echoed to the terminal.
B]CB]--output-directory <path>B]R]
Redirect the C]stdoutR], C]stderrR], and C]stddiagR] of all processes to a process-unique location consisting of [lq]//rank.id/std[out,err,diag][rq], where the C]idR] will be the processes[cq] rank, left-filled with zero[cq]s for correct ordering in listings. Any directories in the filename will automatically be created. A relative path value will be converted to an absolute path based on the current working directory where C]prunR] is executed. Note that this I]will notR] work on environments where the file system on compute nodes differs from that where C]prunR] is executed. This option also supports two case-insensitive directives, specified in comma-delimited form after a colon (C]:R]): C]NOJOBIDR] (omits the jobid directory layer) and C]NOCOPYR] (do not copy the output to the terminal).
B]CB]--stdin <rank>B]R]
The rank of the process that is to receive C]stdinR]. The default is to forward C]stdinR] to rank 0, but this option can be used to forward C]stdinR] to any process. It is also acceptable to specify C]noneR], indicating that no processes are to receive C]stdinR].
B]CB]--merge-stderr-to-stdoutB]R]
Merge C]stderrR] to C]stdoutR] for each process.
B]CB]--map-by :TAGOUTPUTB]R]
Tag each line of output to C]stdoutR], C]stderrR], and C]stddiagR] with C][jobid, MCW_rank]<stdxxx>R] indicating the jobid and rank of the process that generated the output, and the channel which generated it.
B]CB]--map-by :TIMESTAMPOUTPUTB]R]
Timestamp each line of output to C]stdoutR], C]stderrR], and C]stddiagR].
B]CB]--map-by :XMLOUTPUTB]R]
Provide all output to C]stdoutR], C]stderrR], and C]stddiagR] in an xml format.
B]CB]--xterm <ranks>B]R]
Display the output from the processes identified by their ranks in separate C]xtermR] windows. The ranks are specified as a comma-separated list of ranges, with a C]-1R] indicating all. A separate window will be created for each specified process. B]Note:R] C]xtermR] will normally terminate the window upon termination of the process running within it. However, by adding a [lq]![rq] to the end of the list of specified ranks, the proper options will be provided to ensure that C]xtermR] keeps the window open I]afterR] the process terminates, thus allowing you to see the process[cq] output. Each C]xtermR] window will subsequently need to be manually closed. B]Note:R] In some environments, C]xtermR] may require that the executable be in the user[cq]s path, or be specified in absolute or relative terms. Thus, it may be necessary to specify a local executable as [lq]./foo[rq] instead of just [lq]foo[rq]. If C]xtermR] fails to find the executable, C]prunR] will hang, but still respond correctly to a ctrl-c. If this happens, please check that the executable is being specified correctly and try again.

File and Environment Management

To manage files and runtime environment:

B]CB]--path <path>B]R]
C]<path>R] that will be used when attempting to locate the requested executables. This is used prior to using the local C]PATHR] setting.
B]CB]--prefix <dir>B]R]
Prefix directory that will be used to set the C]PATHR] and C]LD_LIBRARY_PATHR] on the remote node before invoking the target process. See the [lq]Remote Execution[rq] section, below.
B]CB]--noprefixB]R]
Disable the automatic C]--prefixR] behavior
B]CB]-s | --preload-binaryB]R]
Copy the specified executable(s) to remote machines prior to starting remote processes. The executables will be copied to the session directory and will be deleted upon completion of the job.
B]CB]--preload-files <files>B]R]
Preload the comma separated list of files to the current working directory of the remote machines where processes will be launched prior to starting those processes.
B]CB]--set-cwd-to-session-dirB]R]
Set the working directory of the started processes to their session directory.
B]CB]--wdir <dir>B]R]
Change to the directory C]<dir>R] before the user[cq]s program executes. See the [lq]Current Working Directory[rq] section for notes on relative paths. B]Note:R] If the C]--wdirR] option appears both on the command line and in an application context, the context will take precedence over the command line. Thus, if the path to the desired working directory is different on the backend nodes, then it must be specified as an absolute path that is correct for the backend node.
B]CB]--wd <dir>B]R]
Synonym for C]--wdirR].
B]CB]-x <env>B]R]
Export the specified environment variables to the remote nodes before executing the program. Only one environment variable can be specified per C]-xR] option. Existing environment variables can be specified or new variable names specified with corresponding values. If multiple C]-xR] options with the same variable name (regardless of value) are provided then the last one listed on the command line will take precedence, and the others will be ignored. The exception to this is for PRTE_MCA_ prefixed environment variables which will report an error in that scenario if any of the values differ. For example: C]$ prun -x DISPLAY -x OFILE=/tmp/out ...R]

The parser for the C]-xR] option is not very sophisticated; it does not even understand quoted values. Users are advised to set variables in the environment, and then use C]-xR] to export (not define) them.

MCA Parameters

Setting MCA parameters take a few different forms depending the target project for the parameter. For example, MCA parameters targeting OpenPMIx will contain the string C]pmixR] in their name, and MCA parameters targeting PRTE will contain the string C]prteR] in their name. See the [lq]MCA[rq] section, below, for finer details on the MCA.

B]CB]--gpmixmca <key> <value>B]R]
Pass global PMIx MCA parameters that are applicable to all application contexts. C]<key>R] is the parameter name; C]<value>R] is the parameter value.
B]CB]--mca <key> <value>B]R]
Send arguments to various MCA modules. See the [lq]MCA[rq] section, below.
B]CB]--pmixmca <key> <value>B]R]
Send arguments to various PMIx MCA modules. See the [lq]MCA[rq] section, below.
B]CB]--prtemca <key> <value>B]R]
Send arguments to various PRTE MCA modules. See the [lq]MCA[rq] section, below.
B]CB]--pmixam <arg0>B]R]
Aggregate PMIx MCA parameter set file list. The C]arg0R] argument is a comma-separated list of tuning files. Each file containing MCA parameter sets for this application context.

Debugging Options

B]CB]--get-stack-tracesB]R]
When paired with the C]--timeoutR] option, C]prunR] will obtain and print out stack traces from all launched processes that are still alive when the timeout expires. Note that obtaining stack traces can take a little time and produce a lot of output, especially for large process-count jobs.
B]CB]--timeout <seconds>B]R]
The maximum number of seconds that C]prunR] will run. After this many seconds, C]prunR] will abort the launched job and exit with a non-zero exit status. Using C]--timeoutR] can be also useful when combined with the C]--get-stack-tracesR] option.

Other Options

There are also other options:

B]CB]--allow-run-as-rootB]R]
Allow C]prunR] to run when executed by the root user (C]prunR] defaults to aborting when launched as the root user).
B]CB]--app <appfile>B]R]
Provide an C]appfileR], ignoring all other command line options.
B]CB]--continuousB]R]
Job is to run until explicitly terminated.
B]CB]--dvm-uriB]R]
Specify the URI of the DVM master, or the name of the file (specified as file:filename) that contains that info.
B]CB]--enable-recoveryB]R]
Enable recovery from process failure [Default = disabled].
B]CB]--disable-recoveryB]R]
Disable recovery (resets all recovery options to off).
B]CB]--map-by :DONOTLAUNCHB]R]
Perform all necessary operations to prepare to launch the application, but do not actually launch it.
B]CB]--index-argv-by-rankB]R]
Uniquely index C]argv[0]R] for each process using its rank.
B]CB]--max-restarts <num>B]R]
Max number of times to restart a failed process.
B]CB]--pidB]R]
PID of the daemon to which we should connect.
B]CB]--report-child-jobs-separatelyB]R]
Return the exit status of the primary job only.
B]CB]--show-progressB]R]
Output a brief periodic report on launch progress.
B]CB]--terminateB]R]
Terminate the DVM.

The following options are useful for developers; they are not generally useful to most users:

B]CB]--map-by :DISPLAYALLOCB]R]
Display a detailed list of the allocation being used by this job.
B]CB]--map-by :DISPLAYDEVELB]R]
Display a more detailed table showing the mapped location of each process prior to launch.
B]CB]--map-by :DISPLAYTOPOB]R]
Display the topology as part of the process map just before launch.
B]CB]--report-state-on-timeoutB]R]
When paired with the C]--timeoutR] command line option, report the run-time subsystem state of each process when the timeout expires.

Description

One invocation of C]prunR] starts an application running under the PRTE DVM. If the application is single process multiple data (SPMD), the application can be specified on the C]prunR] command line.

If the application is multiple instruction multiple data (MIMD), comprising of multiple programs, the set of programs and argument can be specified in one of two ways: Extended Command Line Arguments, and Application Context.

An application context describes the MIMD program set including all arguments in a separate file. This file essentially contains multiple C]prunR] command lines, less the command name itself. The ability to specify different options for different instantiations of a program is another reason to use an application context.

Extended command line arguments allow for the description of the application layout on the command line using colons (C]:R]) to separate the specification of programs and arguments. Some options are globally set across all specified programs (e.g. C]--hostfileR]), while others are specific to a single program (e.g. C]--npR]).

Specifying Host Nodes

Host nodes can be identified on the C]prunR] command line with the C]--hostR] option or in a hostfile. See prte-map(1) for more details.

Application Context or Executable Program?

To distinguish the two different forms, C]prunR] looks on the command line for C]--appR] option. If it is specified, then the file named on the command line is assumed to be an application context. If it is not specified, then the file is assumed to be an executable program.

Locating Files

If no relative or absolute path is specified for a file, C]prunR] will first look for files by searching the directories specified by the C]--pathR] option. If there is no C]--pathR] option set or if the file is not found at the C]--pathR] location, then C]prunR] will search the user[cq]s PATH environment variable as defined on the source node(s).

If a relative directory is specified, it must be relative to the initial working directory determined by the specific starter used. For example when using the rsh or ssh starters, the initial directory is C]$HOMER] by default. Other starters may set the initial directory to the current working directory from the invocation of C]prunR].

Current Working Directory

The C]--wdirR] prun option (and its synonym, C]--wdR]) allows the user to change to an arbitrary directory before the program is invoked. It can also be used in application context files to specify working directories on specific nodes and/or for specific applications.

If the C]--wdirR] option appears both in a context file and on the command line, the context file directory will override the command line value.

If the C]--wdirR] option is specified, C]prunR] will attempt to change to the specified directory on all of the remote nodes. If this fails, C]prunR] will abort.

If the C]--wdirR] option is B]notR] specified, C]prunR] will send the directory name where C]prunR] was invoked to each of the remote nodes. The remote nodes will try to change to that directory. If they are unable (e.g., if the directory does not exist on that node), then C]prunR] will use the default directory determined by the starter.

All directory changing occurs before the user[cq]s program is invoked.

Standard I/O

The PRTE DVM directs UNIX standard input to C]/dev/nullR] on all processes except the rank 0 process. The rank 0 process inherits standard input from C]prunR]. B]Note:R] The node that invoked C]prunR] need not be the same as the node where the rank 0 process resides. PRTE DVM handles the redirection of C]prunR][cq]s standard input to the rank 0 process.

The PRTE DVM directs UNIX standard output and error from remote nodes to the node that invoked C]prunR] and prints it on the standard output/error of C]prunR]. Local processes inherit the standard output/error of C]prunR] and transfer to it directly.

Thus it is possible to redirect standard I/O for applications by using the typical shell redirection procedure on C]prunR].


C]$ prun --np 2 my_app < my_input > my_outputR]Note that in this example I]onlyR] the rank 0 process will receive
the stream from C]my_inputR] on stdin. The stdin on all the other nodes will be tied to C]/dev/nullR]. However, the stdout from all nodes will be collected into the C]my_outputR] file.

Signal Propagation

When C]prunR] receives a C]SIGTERMR] and C]SIGINTR], it will attempt to kill the entire job by sending all processes in the job a C]SIGTERMR], waiting a small number of seconds, then sending all processes in the job a C]SIGKILLR].

C]SIGUSR1R] and C]SIGUSR2R] signals received by C]prunR] are propagated to all processes in the job.

A C]SIGTSTOPR] signal to C]prunR] will cause a C]SIGSTOPR] signal to be sent to all of the programs started by C]prunR] and likewise a C]SIGCONTR] signal to C]prunR] will cause a C]SIGCONTR] sent.

Other signals are not currently propagated by C]prunR].

Process Termination / Signal Handling

During the run of an application, if any process dies abnormally (either exiting before invoking C]PMIx_FinalizeR], or dying as the result of a signal), C]prunR] will print out an error message and kill the rest of the application.

Process Environment

Processes in the application inherit their environment from the PRTE DVM daemon upon the node on which they are running. The environment is typically inherited from the user[cq]s shell. On remote nodes, the exact environment is determined by the boot MCA module used. The C]rshR] launch module, for example, uses either C]rshR]/C]sshR] to launch the PRTE DVM daemon on remote nodes, and typically executes one or more of the user[cq]s shell-setup files before launching the daemon. When running dynamically linked applications which require the C]LD_LIBRARY_PATHR] environment variable to be set, care must be taken to ensure that it is correctly set when booting PRTE DVM.

See the [lq]Remote Execution[rq] section for more details.

Remote Execution

The PRTE DVM requires that the C]PATHR] environment variable be set to find executables on remote nodes. This is typically only necessary in C]rshR]- or C]sshR]-based environments. Batch and scheduled environments typically copy the current environment to the execution of remote jobs, so if the current environment has C]PATHR] and/or C]LD_LIBRARY_PATHR] set properly, the remote nodes will also have it set properly. If the PRTE DVM was compiled with shared library support, it may also be necessary to have the C]LD_LIBRARY_PATHR] environment variable set on remote nodes as well (especially to find the shared libraries required to run user applications).

However, it is not always desirable or possible to edit shell startup files to set C]PATHR] and/or C]LD_LIBRARY_PATHR]. The C]--prefixR] option is provided for some simple configurations where this is not possible.

The C]--prefixR] option takes a single argument: the base directory on the remote node where PRTE DVM is installed. The PRTE DVM will use this directory to set the remote C]PATHR] and C]LD_LIBRARY_PATHR] before executing any user applications. This allows running jobs without having pre-configured the C]PATHR] and C]LD_LIBRARY_PATHR] on the remote nodes.

The PRTE DVM adds the basename of the current node[cq]s [lq]bindir[rq] (the directory where the PRTE DVM[cq]s executables are installed) to the prefix and uses that to set the C]PATHR] on the remote node. Similarly, PRTE DVM adds the basename of the current node[cq]s [lq]libdir[rq] (the directory where the PRTE DVM[cq]s libraries are installed) to the prefix and uses that to set the C]LD_LIBRARY_PATHR] on the remote node. For example:

Local bindir:
/local/node/directory/bin
Local libdir:
/local/node/directory/lib64

If the following command line is used:


C]$ prun --prefix /remote/node/directoryR]The PRTE DVM will add [lq]/remote/node/directory/bin[rq] to the
C]PATHR] and [lq]/remote/node/directory/lib64[rq] to the C]LD_LIBRARY_PATHR] on the remote node before attempting to execute anything.

The C]--prefixR] option is not sufficient if the installation paths on the remote node are different than the local node (e.g., if [lq]/lib[rq] is used on the local node, but [lq]/lib64[rq] is used on the remote node), or if the installation paths are something other than a subdirectory under a common prefix.

Note that executing C]prunR] via an absolute pathname is equivalent to specifying C]--prefixR] without the last subdirectory in the absolute pathname to C]prunR].

For example:


C]$ /usr/local/bin/prun ...R]is equivalent to


C]$ prun --prefix /usr/local ...R]Exported Environment Variables

All environment variables that are named in the form
C]PMIX_[rs]*R] will automatically be exported to new processes on the local and remote nodes. Environmental parameters can also be set/forwarded to the new processes using the MCA parameter C]mca_base_env_listR]. While the syntax of the C]-xR] option and MCA param allows the definition of new variables, note that the parser for these options are currently not very sophisticated - it does not even understand quoted values. Users are advised to set variables in the environment and use the option to export them; not to define them.

Setting MCA Parameters

The C]--mcaR] / C]--pmixmcaR] / C]--prtemcaR] switches (referenced here as [lq]C]--mcaR] switches[rq] for brevity) allow the passing of parameters to various MCA (Modular Component Architecture) modules. MCA modules have direct impact on programs because they allow tunable parameters to be set at run time.

The C]-mcaR] switch takes two arguments: C]<key>R] and C]<value>R]. The C]<key>R] argument generally specifies which MCA module will receive the value. For example, the C]<key>R] [lq]rmaps[rq] is used to select which RMAPS to be used for mapping processes to nodes. The C]<value>R] argument is the value that is passed. For example:

B]CB]prun -prtemca rmaps seq -np 1 fooB]R]
Tells PRTE to use the [lq]seq[rq] RMAPS component, and to run a single copy of [lq]a.out[rq] on an allocated node.

The C]-mcaR] switch can be used multiple times to specify different C]<key>R] and/or C]<value>R] arguments. If the same C]<key>R] is specified more than once, the C]<value>R]s are concatenated with a comma ([lq],[rq]) separating them.

Note that the C]-mcaR] switch is simply a shortcut for setting environment variables. The same effect may be accomplished by setting corresponding environment variables before running C]prunR]. The form of the environment variables depends on the type of the C]--mcaR] switch.

B]CB]--mcaB]R]
C]PRTE_MCA_<key>=<value>R]
B]CB]--pmixmcaB]R]
C]PMIX_MCA_<key>=<value>R]
B]CB]--prtemcaB]R]
[ga]PRTE_MCA_=[ga][ga]

Thus, the C]-mcaR] switch overrides any previously set environment variables. The C]-mcaR] settings similarly override MCA parameters set in the C]$PRTE_PREFIX/etc/prte-mca-params.confR] or C]$HOME/.prte/mca-params.confR] file.

Unknown C]<key>R] arguments are still set as environment variable [en] they are not checked (by C]prunR]) for correctness. Illegal or incorrect C]<value>R] arguments may or may not be reported [en] it depends on the specific MCA module.

To find the available component types under the MCA architecture, or to find the available parameters for a specific component, use the C]pinfoR] command. See the I]pinfo(1)R] man page for detailed information on the command.

Running as root

The PRTE team strongly advises against executing C]prunR] as the root user. Applications should be run as regular (non-root) users.

Reflecting this advice, C]prunR] will refuse to run as root by default. To override this default, you can add the C]--allow-run-as-rootR] option to the C]prunR] command line.

Return Value

There is no standard definition for what C]prunR] should return as an exit status. After considerable discussion, we settled on the following method for assigning the C]prunR] exit status (note: in the following description, the [lq]primary[rq] job is the initial application started by C]prunR] - all jobs that are spawned by that job are designated [lq]secondary[rq] jobs):

[bu]
if all processes in the primary job normally terminate with exit status 0, we return 0
[bu]
if one or more processes in the primary job normally terminate with non-zero exit status, we return the exit status of the process with the lowest rank to have a non-zero status
[bu]
if all processes in the primary job normally terminate with exit status 0, and one or more processes in a secondary job normally terminate with non-zero exit status, we (a) return the exit status of the process with the lowest rank in the lowest jobid to have a non-zero status, and (b) output a message summarizing the exit status of the primary and all secondary jobs.
[bu]
if the cmd line option C]--report-child-jobs-separatelyR] is set, we will return -only- the exit status of the primary job. Any non-zero exit status in secondary jobs will be reported solely in a summary print statement.

By default, the job will abort when any process terminates with non-zero status. The MCA parameter C]prte_abort_on_non_zero_statusR] can be set to C]falseR] (or C]0R]) to cause the PRTE DVM to not abort a job if one or more processes return a non-zero status. In that situation the PRTE DVM records and notes that processes exited with non-zero termination status to report the approprate exit status of C]prunR] (per bullet points above).

If the C]--timeoutR] command line option is used and the timeout expires before the job completes (thereby forcing C]prunR] to kill the job) C]prunR] will return an exit status equivalent to the value of C]ETIMEDOUTR] (which is typically 110 on Linux and OS X systems).


Table of Contents

« Return to documentation listing