Uploading Files

Uploading Files#

Overview#

Materials Commons offers three different methods for uploading files, each suited for different needs:

1. Globus Upload#

Best for: Large datasets and reliable transfers

Key Features:#

  • Secure and reliable transfer for large datasets

  • Automatic retry and error recovery

  • Progress monitoring and email notifications

  • Supports resume of interrupted transfers

Requirements:#

  • MaterialsCommons account

  • Globus account

  • Globus Connect Personal client installed on computer you are uploading from

2. Web File Uploader#

Best for: Small to medium files and quick uploads

Key Features:#

  • Direct browser-based upload

  • Single and multiple file upload support

  • Directory upload capability

  • Real-time progress monitoring

  • Maximum file size: 250MB per file

Requirements:#

  • Modern web browser (Chrome, Firefox, Safari, or Edge)

  • MaterialsCommons account

  • Stable internet connection

3. CLI File Uploader#

Best for: Automated uploads and scripted operations

Key Features:#

  • Command-line interface for uploads

  • Supports recursive directory uploads

  • Pattern-based file inclusion/exclusion

  • Dry-run capability for testing

  • Project management functions

Requirements:#

  • Python and pip installed

  • MaterialsCommons API key

  • Command-line access

Note

Choose your upload method based on: * File size and quantity * Transfer reliability needs * Automation requirements * Available tools and infrastructure

For more detailed information on each method, refer to the specific documentation sections.

Using Globus to Upload Data#

Overview#

Globus provides a reliable and secure way to transfer large datasets to MaterialsCommons. This guide will walk you through the process of setting up and using Globus for data transfers.

Prerequisites#

Before you begin, ensure you have:

  • A MaterialsCommons account

  • A Globus account (sign up at globus.org)

  • The Globus Connect Personal client installed (if transferring from a personal computer)

Setting Up Globus Connect Personal#

  1. Download and Install

    • Visit the Globus Connect Personal page

    • Download the version for your operating system

    • Follow the installation instructions for your platform

  2. Configure Globus Connect Personal

    • Launch the application

    • Sign in with your Globus account

    • Set up your endpoint name (this is how your computer will be identified in Globus)

    • Configure access to your local directories

Connecting to MaterialsCommons via Globus#

  1. Log into MaterialsCommons

    • Navigate to your project

    • To upload click on “Globus Upload” in the sidebar

      • Materials Commons will create a link to take you to Globus. This will be where you upload your files to

      • When you are done uploading you will have to tell Materials Commons you are finished so it can finish loading the files into your project

        • Click on “Globus Upload”, click “Done” when you are finished transferring files

  2. Access Globus File Manager

    • Go to app.globus.org

    • Sign in with your Globus account

    • Click on “File Manager”

Starting a Transfer#

  1. Set Up Source Location

    • In the Globus File Manager, click on “Collection” in the left panel

    • Select your source endpoint (your local computer or another collection)

    • Navigate to the directory containing your data

  2. Set Up Destination Location

    • Click on “Collection” in the right panel

    • Search for “MaterialsCommons” or use the endpoint ID

    • Navigate to your project directory

  3. Initiate Transfer

    • Select the files/folders you want to transfer

    • Click the “Start” button between the panels

    • Verify transfer settings

    • Click “Submit”

Monitoring Transfers#

  1. Check Transfer Status

    • Click on “Activity” in the main menu

    • View current and past transfers

    • Monitor progress and check for any errors

  2. Transfer Notifications

    • Set up email notifications (optional)

    • Receive alerts when transfers complete or if errors occur

Best Practices#

  1. Directory Structure

    • Organize files before transfer (see data organization guide)

    • Maintain consistent naming conventions

    • Keep directory paths as short as possible

  2. File Considerations

    • Compress large collections of small files before transfer

    • Consider breaking up extremely large files

    • Verify file permissions

  3. Transfer Settings

    • Enable verification for important transfers

    • Use sync options for updating existing directories

    • Set appropriate concurrency based on file sizes

Troubleshooting#

Common Issues#

  1. Connection Problems

    • Check your internet connection

    • Verify endpoint is online

    • Ensure Globus Connect Personal is running (if using)

  2. Permission Errors

    • Check file and folder permissions

    • Verify you have access to the MaterialsCommons project

    • Confirm endpoint authorization

  3. Transfer Failures

    • Check available storage space

    • Verify file names are compatible

    • Look for firewall or network restrictions

Getting Help#

If you encounter issues:

  • Check the Globus status page

  • Contact MaterialsCommons support

  • Review Globus documentation

  • Submit a support ticket

Security Considerations#

  1. Data Protection

    • Globus uses encryption for all transfers

    • Keep your credentials secure

    • Log out when finished

  2. Access Control

    • Review sharing settings regularly

    • Remove unused endpoint access

    • Monitor transfer activity

Tips for Large Transfers#

  1. Planning

    • Estimate transfer time and space requirements

    • Schedule transfers during off-peak hours

    • Break large transfers into smaller batches

  2. Optimization

    • Use appropriate concurrency settings

    • Enable verification for critical data

    • Monitor system resources

  3. Verification

    • Compare file counts and sizes

    • Verify checksums for critical data

    • Document successful transfers

Note

Always verify that your transfers completed successfully before deleting source files.

Warning

Never interrupt a transfer in progress unless absolutely necessary. Use the Globus interface to cancel transfers properly.

Additional Resources#

Tip

Keep your Globus Connect Personal client updated to ensure the best performance and security.

Using The Materials Commons Web File Uploader#

Overview#

The MaterialsCommons web file uploader provides a convenient way to upload files and directories directly through your web browser. This guide covers both single/multiple file uploads and directory uploads.

Prerequisites#

  • A modern web browser (Chrome, Firefox, Safari, or Edge)

  • A MaterialsCommons account with access to a project

  • Stable internet connection

  • Files organized and ready for upload

File Upload#

Single File Upload#

  1. Navigate to Upload Location

    • Go to your project in MaterialsCommons

    • Browse to the directory where you want to upload

    • Click the “Upload” button

  2. Select File

    • Click “browse files” or drag files into the upload area

    • To upload a directory and everything under it click “browse folders”

    • Select the file you want to upload

    • Verify the file name and size

  3. Initiate Upload

    • Click “Upload” to start the transfer

    • Wait for the progress bar to complete

    • Confirm the file appears in your directory

Multiple File Upload#

  1. Select Multiple Files

    • Click “Choose Files” or drag multiple files

    • Hold Ctrl (Windows) or Command (Mac) to select multiple files

    • Alternatively, drag a selection box over files

    • Review selected files in the upload queue

  2. Upload Settings

    • Verify destination directory

    • Check total size and file count

    • Review file names for any conflicts

  3. Start Upload

    • Click “Upload” to begin

    • Monitor overall progress

    • Wait for all files to complete

Directory Upload#

Preparing for Directory Upload#

  1. Browser Requirements

    • Ensure your browser supports directory upload

    • Chrome and Firefox are recommended

    • Update your browser if needed

  2. Directory Structure

    • Organize files into desired structure

    • Check for any unsupported file types

    • Remove any unnecessary hidden files

Uploading a Directory#

  1. Select Directory

    • Click “Upload Directory” button

    • Choose directory from your computer

    • Or drag entire folder into upload area

  2. Review Selection

    • Check directory name and structure

    • Verify number of files

    • Confirm total size

  3. Start Upload

    • Click “Upload” to begin transfer

    • Monitor progress indicator

    • Wait for completion message

Managing Uploads#

Progress Monitoring#

  1. Upload Status

    • View overall progress percentage

    • See individual file progress

    • Check estimated time remaining

    • Monitor upload speed

  2. Status Indicators

    • ● Green - Successfully uploaded

    • ● Yellow - In progress

    • ● Red - Failed or error

Handling Errors#

  1. Common Issues

    • Network interruption

    • File too large

    • Name conflicts

    • Permission issues

  2. Resolution Steps

    • Check internet connection

    • Verify file size limits

    • Resolve naming conflicts

    • Check project permissions

Best Practices#

File Preparation#

  1. Before Upload

    • Organize files logically

    • Use clear naming conventions

    • Remove special characters from names

    • Check file sizes

  2. During Upload

    • Keep browser tab active

    • Avoid network changes

    • Don’t close browser

    • Monitor available space

Upload Strategies#

  1. Large Files

    • Upload larger files first

    • Consider breaking into smaller parts

    • Use wired internet connection

    • Allow sufficient time

  2. Many Small Files

    • Use directory upload when possible

    • Group related files together

    • Consider creating archives

    • Monitor system resources

Limitations and Considerations#

  1. Size Limits

    • Maximum file size: 250MB per file

    • Total upload size varies by project

  2. Browser Limitations

    • Keep browser tab active

    • Avoid multiple simultaneous uploads

    • Clear cache if issues occur

  3. File Types

    • All file types supported

    • Some extensions may be restricted

    • Special handling for certain types

Troubleshooting#

Common Problems#

  1. Upload Fails to Start

    • Clear browser cache

    • Check file permissions

    • Verify internet connection

    • Try a different browser

  2. Upload Interruption

    • Restart failed uploads

    • Check network stability

    • Verify browser is up-to-date

    • Contact support if persistent

Recovery Options#

  1. Failed Uploads

    • Retry individual files

    • Check error messages

    • Verify file integrity

    • Contact support if needed

Tips for Success#

  1. Performance

    • Use Chrome or Firefox for best results

    • Close unnecessary browser tabs

    • Maintain stable internet connection

    • Upload during off-peak hours

  2. Organization

    • Plan directory structure beforehand

    • Use meaningful file names

    • Keep path names short

    • Document upload process

Note

Large uploads may take significant time. Plan accordingly and ensure your computer won’t go to sleep during upload.

Warning

Don’t close your browser or navigate away from the page during active uploads.

Tip

For very large transfers, consider using Globus instead of the web uploader.

Getting Help#

If you encounter issues:

  • Check the MaterialsCommons documentation

  • Contact project administrator

  • Submit support ticket

  • Check browser console for errors

Additional Resources#

Using The Materials Commons CLI File Uploader#

Installation#

Install the Materials Commons CLI tool using pip:

pip install materials_commons_cli

Initial Setup#

  1. Generate an API key from Materials Commons website

  2. Configure the CLI with your API key:

mc config

Project Setup#

Create a new project:

mc project create "Project Name"

List existing projects:

mc projects

Set current working project:

mc project use <project_id>

File Upload Commands#

Upload a single file:

mc upload <file_path>

Upload a directory:

mc upload -r <directory_path>

Upload with specific options:

mc upload --include "*.txt" --exclude "*.tmp" <directory_path>

Common Options#

Option

Description

-r, –recursive

Upload directories recursively

–include

Pattern for files to include

–exclude

Pattern for files to exclude

–dry-run

Show what would be uploaded without actually uploading

Project Management#

Check project status:

mc project status

List project files:

mc files list

Create directory in project:

mc files mkdir <directory_name>

Best Practices#

  1. Always verify your current project before uploading:

    mc project current

  2. Use --dry-run before large uploads:

    mc upload --dry-run -r <directory>

  3. Organize files into directories before upload:

    mc files mkdir data
mc files mkdir results
mc upload -r ./local-data data/

Note

Remember to check upload status and verify files after upload completion. Use mc files list to confirm successful uploads.

Troubleshooting#

If uploads fail:

  1. Check your API key configuration

  2. Verify project access permissions

  3. Ensure stable internet connection

  4. Try uploading smaller batches

For more detailed information, consult the Materials Commons CLI documentation.

Contents