pegasus-transfer(1)
===================
:doctype: manpage


Name
----
pegasus-transfer - Handles data transfers in Pegasus workflows.


Synopsis
--------
[verse]
*pegasus-transfer* [*-h*]
                   [*--file* 'inputfile']
                   [*--threads* 'number_threads']
                   [*--max-attempts* 'attempts']
                   [*--debug*] 


Description
-----------
*pegasus-transfer* takes a list of url pairs, either on stdin or with 
an input file, determines the correct tool to use for the transfer and 
executes the transfer. Some of the protocols pegasus-transfer can handle 
are GridFTP, SRM, Amazon S3, HTTP, and local cp/symlinking. Failed 
transfers are retried.

Note that pegasus-transfer is a tool mostly used internally in Pegasus
workflows, but the tool can be used stand alone as well.

Options
-------
*-h*::
*--help*::
Prints a usage summary with all the available command-line options.

*-f* 'inputfile'::
*--file* 'inputfile'::
File with input pairs. If not given, stdin will be used.

*-m*::
*--max-attempts* 'attempts'::
Maximum number of attempts for retrying failed transfers.

*-t*::
*--threads* 'number_threads'::
The number of threads to use. This controls the parallelism of transfers. 

*-d*::
*--debug*::
Enables debugging output.

Example
-------
-----------
$ pegasus-transfer
# src 1 local_site
file:///etc/hosts
# dst 1 local_site
file:///tmp/foo
CTRL+D
-----------

Credential Handling
-------------------

Credentials used for transfers can be specified with a combination of
comments in the input file format and environment variables. For example,
give the following input file:

-----------
# src 1 isi
gsiftp://workflow.isi.edu/data/file.dat
# dst 1 tacc_stampede
gsiftp://gridftp.stampede.tacc.utexas.edu/scratch/file.dat
-----------
 
pegasus-transfer will expect either one environment variable specifying
one credential to be used on both end of the connection (X509_USER_PROXY),
or two separate environment variables specifying two different credentials
to be used on the two ends of the connection. The the latter case, the
environment variables are derived from the site names provided in the
input file commments. In the example above, the environment variables
would be named X509_USER_PROXY_isi and X509_USER_PROXY_tacc_stampede

Threading
---------

In order to speed up data transfers, pegasus-transfer will start a set
of transfers in parallel using threads. Threads are turned off when
retrying failed transfers.


Author
------
Pegasus Team <http://pegasus.isi.edu>
