Interacting with 3DEM.org

Known Issues and Limitations

• When files are copied or moved on 3DEM.org from a browser, it will only indicate if the operation succeeded or failed. It does not show the progress.
  • Workaround: You can monitor the progress by accessing lonestar6, stampede2, and corral via WinSCP (or equivalent, or terminal).
• When files are transferred from lonestar6/stampede2 to corral via command lines, those files become inaccessible on 3DEM.org from a browser. You can see the folder exists, but when you try to open it, you get the following message:  The request could not be understood by the server due to malformed syntax.
  
  • Why: This is because the web interface of 3DEM.org needs permissions to access them.
  • Solution: Submit a ticket on 3DEM.org or contact James Carson, who can show you how to set the correct permissions.
  • How can I avoid this:
    • Solution A: (1) Copy (not move) your files from lonestar6/stampede2 to corral via 3DEM.org portal, (2) monitor the progress by accessing lonestar6/stampede2 and corral via WinSCP (or equivalent, or terminal), and (3) once completed, delete the source files on lonestar6/stampede2.
    • Solution B: (1) Set up a shared workspace on 3dem.org on a browser, and (2) Copy (not move) your files from lonestar6/stampede2 to corral via command line.

Multi-factor Authentication at TACC

After setting up your 3dem.org account, you need to set up two-factor authentication. See instructions here: https://portal.tacc.utexas.edu/tutorials/multifactor-authentication

Where are My Files on 3dem.org?

"My Data"

• "My Data" shows your work directory on a TACC supercomputer (e.g., Stampede2, Lonestar6, etc.).
• The full file path of your work directory should look like: /work/#####/username/stampede2, where ##### = 5-digit number assigned to you, and username = your TACC username.
• Your work directory has a disk quota of 1TB.
• If you have been using /work, switch to /work2. /work will be come read-only on April 19, 2021and /work2 will be the only work file system on May 17, 2021.

"Shared Workspaces"

• "Shared Workspaces" shows project folders you have access to.
• The project folders are located on Corral (data.tacc.utexas.edu), which is a georeplicated file server.
• A 4-digit Project ID is assigned to each project folder. The full file path of a project folder would look like: /corral-repl/projects/NeuroNex-3DEM/projects/3dem-####, where #### = project ID.

How to create a new shared workspace

This should be done by a person responsible for the project/data to be stored under the new folder.

1. Log into 3DEM.org and navigate to Workbench > Data Files > Shared Workspace
2. Click Add > Shared Workspace
3. Type in a title of the new project folder (e.g., Interneuron) and a short description, then click Create Project.
4. Assign PI to the new folder. This should be the person in charge of the project (most likely you). If you assign someone else as PI, then you will not have access to this folder until the PI gives you permissions.
5. Assign Co-PIs to the new folder. This typically includes Kristen Harris, James Carson, and anyone who should be able to administer the data.
6. Add Members. This would be other members of the project.

• PI. Co-PI, and Members all must have 3DEM.org account.

Copy files from a project folder to your work directory (or vice versa)

1. Click on folder icon of the folder you want to copy.
2. Click Copy
3. Navigate to the destination and click Copy to ... button.
4. Verify the transfer. You could do this with terminal or a file transfer tool like WinSCP and FileZilla.

• Use Copy, not Move, so that you wouldn't lose data if something goes wrong during transfer. If you are permanently transferring data from your work directory to a project folder, verify that data transfer is complete, then delete the source.

How do I transfer files to TACC?

• See here for using Globus.
• See here for using command lines.
• Also see How to transfer files/folders between khbackup and stampede2/corral (Harris lab specific; restricted access)