[gt-user] directory transfers in RSL

[Adam Bazinet]

Dear Globus users/admins,

We've been trying to get directory transfers to work with RSL in GT6
(GRAM5), but haven't had any luck so far. We haven't been able to find any
examples online of RSL that includes a directory transfer as part of
file_stage_in or file_stage_out, but we thought it might be possible
because it does work in our GT4-based system.

Here is an example attempted directory transfer:

(file_stage_in = (gsiftp://
arginine.umiacs.umd.edu/home/gt6admin/GT6Upgrade/GT4.2/garli/trunk/test/emptyDir/
$(HOME)/2535472060.225141373834153/))

Note that $(HOME)/2535472060.225141373834153/ does not exist before this
job is executed --- transferring an empty directory and essentially
renaming it is the trick we use with GT4 to cause the working directory to
be created remotely before we stage additional files into it. However, we
have also tried non-empty directories, and those don't work either.

So anyway, the question's pretty simple: can you transfer entire
directories in GT6-compatible RSL? As an aside, we can get the behavior we
want out of globus-url-copy by adding the -cd and -r flags, but it's
unclear if this functionality is available through GRAM.

thanks,
Adam

[Stuart Martin]

Hi Adam,

Unfortunately, GRAM5 does not support creating a new directory as part of the file stage in or out, except via the scratch_dir RSL attribute. At present, stage in and out only support a list of file paths and not directories.

Do you have a sufficient workaround? Is it working for you to make and remove the job directory outside of GRAM, e.g. before and after the GRAM job submission?

Regards,
Stu

Post by Adam Bazinet
Dear Globus users/admins,
We've been trying to get directory transfers to work with RSL in GT6 (GRAM5), but haven't had any luck so far. We haven't been able to find any examples online of RSL that includes a directory transfer as part of file_stage_in or file_stage_out, but we thought it might be possible because it does work in our GT4-based system.
(file_stage_in = (gsiftp://arginine.umiacs.umd.edu/home/gt6admin/GT6Upgrade/GT4.2/garli/trunk/test/ <http://arginine.umiacs.umd.edu/home/gt6admin/GT6Upgrade/GT4.2/garli/trunk/test/>emptyDir/ $(HOME)/2535472060.225141373834153/))
Note that $(HOME)/2535472060.225141373834153/ does not exist before this job is executed --- transferring an empty directory and essentially renaming it is the trick we use with GT4 to cause the working directory to be created remotely before we stage additional files into it. However, we have also tried non-empty directories, and those don't work either.
So anyway, the question's pretty simple: can you transfer entire directories in GT6-compatible RSL? As an aside, we can get the behavior we want out of globus-url-copy by adding the -cd and -r flags, but it's unclear if this functionality is available through GRAM.
thanks,
Adam

[Adam Bazinet]

Hi Stu,

Thanks for the information. We should be able to work around this
limitation going forward, but it would have certainly simplified our
GT4->GT6 upgrade if this functionality had been present.

best,
Adam

Post by Stuart Martin
Hi Adam,
Unfortunately, GRAM5 does not support creating a new directory as part of
the file stage in or out, except via the scratch_dir RSL attribute. At
present, stage in and out only support a list of file paths and not
directories.
Do you have a sufficient workaround? Is it working for you to make and
remove the job directory outside of GRAM, e.g. before and after the GRAM
job submission?
Regards,
Stu
Dear Globus users/admins,
We've been trying to get directory transfers to work with RSL in GT6
(GRAM5), but haven't had any luck so far. We haven't been able to find
any examples online of RSL that includes a directory transfer as part of
file_stage_in or file_stage_out, but we thought it might be possible
because it does work in our GT4-based system.
(file_stage_in = (gsiftp://
arginine.umiacs.umd.edu/home/gt6admin/GT6Upgrade/GT4.2/garli/trunk/test/emptyDir/
$(HOME)/2535472060.225141373834153/))
Note that $(HOME)/2535472060.225141373834153/ does not exist before this
job is executed --- transferring an empty directory and essentially
renaming it is the trick we use with GT4 to cause the working directory to
be created remotely before we stage additional files into it. However, we
have also tried non-empty directories, and those don't work either.
So anyway, the question's pretty simple: can you transfer entire
directories in GT6-compatible RSL? As an aside, we can get the behavior we
want out of globus-url-copy by adding the -cd and -r flags, but it's
unclear if this functionality is available through GRAM.
thanks,
Adam