Using The CLI

Using The CLI#

Introduction#

This guide covers using the Materials Commons CLI. The CLI provides an efficient terminal-based workflow.

Installation#

Before getting started, ensure Materials Commons is properly installed:

Note

It is recommended that you install using –user that the package is local to your account:

The --user option will install to the Python user install directory for your platform. Typically ~/.local/, or %APPDATA%Python on Windows.

Install the Materials Commons CLI:
1. Install the CLI locally to your user account (recommended):
```
pip install --user materials-commons-cli
```
Note

When installed locally make sure to add the installation directory to your path:
```
export PATH="$PATH:$(python3 -m site --user-base)/bin"
```
1. Install the CLI globally:
```
sudo pip install materials-commons-cli
```

Adding Your API Key#

The Materials Commons CLI requires authentication through an API key to interact with a Materials Commons instance (called a “remote”). This guide explains how to configure your CLI access.

Configuration Steps#

Configure Remote Instance

Set up the CLI to connect to a specific Materials Commons server (e.g., materialscommons.org)

mc remote –add <email> https://materialscommons.org/api
Set Default Remote

Establish your primary Materials Commons instance for CLI operations

mc remote –set-default <email> https://materialscommons.org/api

Note

You will be prompted for your password to retrieve and save your API key locally.

Warning

Security Notice - API Key Storage

The Materials Commons CLI stores authentication credentials in a configuration file:

$HOME/.materialscommons/config.json

Security Implications:

This file contains sensitive API keys
Unauthorized access to these keys enables full account access

Never share or expose this configuration file

Project Setup#

Before starting your project setup, we recommend reviewing the best practices guide for optimal file organization.

Important

When using the Materials Commons CLI, all project files must reside in a single root directory. This directory serves as the base location that will be linked to your Materials Commons project.

Creating a New Project#

Note

After organizing your data according to the previous guidelines, you can create and link a new Materials Commons project using the CLI. The following steps will guide you through this process.

Create and navigate to a new directory:
```
mkdir <project-name>
cd <project-name>
```
Initialize the project:
```
mc init
```

Note

The project will be created with the same name as your directory.

Using an Existing Project#

Use the mc clone command to create a local copy of your existing Materials Commons project.

Important

The cloned project directory will appear empty initially. Files from the remote project are not automatically downloaded - they must be explicitly requested through download commands.

Create and navigate to a new directory:
```
mkdir <project-name>
cd <project-name>
```

List available projects:

All existing Materials Commons projects you have access to can be listed using mc proj:

$ mc proj
    name                       owner                  id    updated_at
--  -------------------------  ---------------------  ----  --------------------
    Mg-Nd CASM project         adam@email.com         33    2020 Oct 16 17:40:12
    Mg-Y CASM project          adam@email.com         39    2020 Oct 16 17:40:12

Clone the desired project:
```
mc clone <project-id>
```

Note

The project ID can be found in the ‘id’ column of the mc proj output.

File Upload/Download#

The mc command line interface enables three primary operations between your local project and Materials Commons:

Upload files and changes
Download files and changes
Synchronize content bidirectionally * This happens when you upload/download recursively

Note

A local project directory must exist (created using mc init or mc clone) before transferring files.

Basic Commands#

Upload files: mc up [-r] local_file [local_file ...]
Download files: mc down [-r] remote_file [remote_file ...]

Important

The -r option enables recursive directory transfers
Missing intermediate directories are created automatically
Files are transferred to corresponding paths in remote/local projects

Example Directory Structures#

Remote project:

MyProject/
├── file_A.txt
├── file_B.txt
└── level_1/
    ├── file_1A.txt
    ├── file_1B.txt
    └── level_2/
        ├── file_2A.txt
        └── file_2B.txt

Local project:

MyProject/
├── file_A.txt
├── file_C.txt
└── level_1/
    ├── file_1A.txt
    └── file_1C.txt

Common Usage Examples#

Note

The command examples below assume you are in the MyProject directory.

Upload Operations:

Single file: mc up file_C.txt
Multiple files: mc up file_C.txt level_1/file_1C.txt
Directory recursively: mc up -r level_1
Everything recursively: mc up -r .

Download Operations:

Single file: mc down file_B.txt
Multiple files: mc down file_B.txt level_1/file_1B.txt
With directory creation: mc down level_1/level_2/file_2A.txt
Directory recursively: mc down -r level_1
Everything recursively: mc down -r .

Warning

There is a default upload size limit of 250MB per file. Files exceeding this limit will be skipped.

Note

Files are compared against MD5 checksums by default
Use --no-compare to skip checksum verification
Uploaded files are automatically versioned on Materials Commons
mc down will not overwrite files with changes that haven’t been uploaded.
- To force an overwrite use the mc down --force command

Uploading Files with Globus#

Note

Globus transfers use a temporary intermediary directory on Materials Commons.

Upload Process Overview#

1. Create Upload Directory#

$ mc globus upload --create
Created Globus upload: 293
    project_name      project_id  type    name                   id  uuid                                  created               status
--  --------------  ------------  ------  -------------------  ----  ------------------------------------  --------------------  --------
*   MyProject                589  upload  rivage-popish-bonze   293  c3474f16-b357-4b6c-92ca-fd93a1e37840  2021 Jan  1 03:52:49  Ready

Note

Creates an empty upload directory with a unique name (three random words + ID + UUID)
Multiple upload directories can exist simultaneously
Access is private to your account only

2. Transfer Files#

Important

Use the -g or --globus option with mc up commands:

Single file: mc up -g file.txt
Multiple files: mc up -g file1.txt file2.txt
Recursive upload: mc up -g -r directory/
Upload all: mc up -g -r .

Example of first-time transfer:

$ mc up -g file_B.txt
Using current globus upload (name=rivage-popish-bonze, id=293).
Please login. If a webpage does not open automatically, go here:

https://auth.globus.org/v2/oauth2/authorize?... (long link)

Please enter the code you get after login here: abc123abc123abc123abc123abc123
Globus task_id: c127a968-57b1-11eb-87bb-02187389bd35
Globus transfer task initiated.
Use `globus task list` to monitor task status.
Use `mc globus upload` to manage Globus uploads.

Check transfer status:

$ globus task list
Task ID                              | Status    | Type     | Source Display Name | Dest Display Name          | Label
------------------------------------ | --------- | -------- | ------------------- | -------------------------- | --------------------------------
c127a968-57b1-11eb-87bb-02187389bd35 | SUCCEEDED | TRANSFER | my_MacbookAir       | materials-commons-2-upload | MyProject-rivage-popish-bonze

Tip

Files mirror the project’s directory structure
Only needed directories are created
Multiple transfers can run simultaneously
Use --goto for web interface: mc globus upload --id <ID> --goto

3. Finish Upload#

$ mc globus upload --id 293 --finish

Check status:

$ mc globus upload
    project_name      project_id  type    name                   id  uuid                                  created               status
--  --------------  ------------  ------  -------------------  ----  ------------------------------------  --------------------  ---------
    MyProject                589  upload  rivage-popish-bonze   293  c3474f16-b357-4b6c-92ca-fd93a1e37840  2021 Jan 16 03:52:49  Finishing

Managing Upload Directories#

Important

Key management commands:

Delete without processing: mc globus upload --delete
Set current directory: mc globus upload --set
Unset current directory: mc globus upload --unset
Check MD5 checksums automatically for changed files

Downloading Files with Globus#

Initialize download:

$ mc globus download --create
Created Globus download: 295
    project_name      project_id  type      name                   id  uuid                                  created               status
--  --------------  ------------  --------  -------------------  ----  ------------------------------------  --------------------  --------
*   MyProject                589  download  teerer-armed-gynics   295  db89ab27-92cf-4e12-9b4d-a4c3162f56da  2021 Jan 16 04:52:24  Waiting

Warning

Known issue: Creating download directories for projects with many files may be slow or fail. Improved method in development.

Check status:

$ mc globus download
    project_name      project_id  type      name                   id  uuid                                  created               status
--  --------------  ------------  --------  -------------------  ----  ------------------------------------  --------------------  --------
*   MyProject                589  download  teerer-armed-gynics   295  db89ab27-92cf-4e12-9b4d-a4c3162f56da  2021 Jan 16 04:52:24  Waiting

Download Process#

Initialize: * Creates temporary download directory * Populates with links to project files * Private access control
Transfer: * Basic download: mc down -g file_B.txt * Recursive downloads: mc down -g -r directory/ * Web interface: mc globus download --id <ID> --goto
Cleanup:

mc globus download --id 295 --delete

Note

Directory persists until deleted
Won’t reflect project changes after creation
Manage download directories with --set and --unset options

File Operations#

Important

These operations require a local project directory created via mc init or mc clone

Available Commands#

Command	Purpose
`mc ls`	List files and directories
`mc mkdir`	Create directories
`mc rm`	Remove files/directories
`mc mv`	Move/rename files/directories

Note

Commands affect both local and remote files by default
Use --remote-only to operate on remote files only

Listing Files (mc ls)#

Quick Reference#

mc ls [-h] [--checksum] [--json] [--dataset DATASET_ID] [--include] [--exclude] [--clear] [paths ...]

Tip

The mc ls command shows both local and remote information:

Local entries prefixed with l_
Remote entries prefixed with r_
Use --checksum to compare file contents via MD5

Example Output#

$ mc ls . level_1 --checksum
.:
l_updated_at          l_size    l_type     r_updated_at          r_size    r_type     eq    name        id
--------------------  --------  ---------  --------------------  --------  ---------  ----  ----------  -------
2020 Aug 17 23:25:48  23B       file       2020 Aug 18 03:59:05  23B       file       True  file_A.txt  2659322
2020 Aug 17 23:25:48  23B       file       -                     -         -          -     file_B.txt  -
2020 Aug 17 23:25:48  -         directory  2020 Aug 18 03:31:46  0B        directory  -     level_1     2659316

level_1:
l_updated_at          l_size    l_type     r_updated_at          r_size    r_type     eq    name             id
--------------------  --------  ---------  --------------------  --------  ---------  ----  ----------  -------
2020 Aug 17 23:25:48  31B       file       2020 Aug 18 03:31:47  31B       file       True  file_A.txt  2659317
2020 Aug 17 23:25:48  31B       file       2020 Aug 18 03:31:49  31B       file       True  file_B.txt  2659318
2020 Aug 17 23:25:48  -         directory  2020 Aug 18 03:31:49  0B        directory  -     level_2     2659319

Understanding the Output#

Column	Description
updated_at	Timestamp of last modification
size	File size (B = bytes)
type	file or directory
eq	True if local and remote checksums match
id	Remote file identifier

Warning

“-” in remote columns indicates file/directory exists only locally
Missing checksum (eq) for directories is normal
File sizes may differ between local and remote copies

Making Directories (mc mkdir)#

Quick Reference#

mc mkdir [-h] [-p] [--remote-only] [paths ...]

Description#

Creates directories in both local and remote project locations.

Note

Creates directories specified in paths
By default, operates on both local and remote locations
Fails if parent directories don’t exist (unless -p is used)

Options#

Option	Description	Default
`paths`	Directory names to create	Current directory
`-p`	Create parent directories if needed	`False`
`--remote-only`	Create directories only on remote	`False`

Examples#

# Create single directory
mc mkdir new_dir

# Create nested directories
mc mkdir -p path/to/new/dir

# Create multiple directories
mc mkdir dir1 dir2 dir3

# Create remote-only directory
mc mkdir --remote-only remote_dir

Warning

Without -p, fails if parent directories don’t exist
--remote-only skips local directory creation
Directory names should follow filesystem naming conventions

Tip

Use -p option when creating nested directory structures to ensure all parent directories are created automatically

Moving Files/Directories (mc mv)#

Quick Reference#

mc mv [-h] [--remote-only] [paths ...]

Description#

Moves or renames files and directories in both local and remote locations. Supports two operation modes:

Single move/rename: mc mv <source> <target>
Multiple move: mc mv <source1> <source2> ... <destination_dir>

Note

Acts on both local and remote locations by default
Can move single or multiple items
Target directory must exist for multiple moves

Options#

Option	Description	Default
`paths`	Source and destination paths	Required
`--remote-only`	Move only remote files/directories	`False`

Examples#

# Rename file
mc mv old_name.txt new_name.txt

# Move single file to directory
mc mv file.txt target_directory/

# Move multiple files to directory
mc mv file1.txt file2.txt directory/

# Move remote files only
mc mv --remote-only source.txt target_directory/

Warning

Target directory must exist for multiple file moves
Moving directories will move all contents
--remote-only affects only remote files, leaving local files unchanged

Tip

Use trailing slash for directories to make intentions clear
Check paths before moving multiple files
Use mc ls to verify results after moving

Deleting Files/Directories (mc rm)#

Quick Reference#

mc rm [-h] [-r] [--remote-only] [--no-compare] [paths ...]

Description#

Deletes files and directories from both local and remote locations.

Important

By default, performs safety checks:

Requires -r flag for directory removal
Verifies local/remote file consistency
Affects both local and remote locations

Options#

Option	Description	Default
`paths`	Files/directories to remove	Required
`-r, --recursive`	Remove directories and their contents	`False`
`--remote-only`	Remove only from remote location	`False`
`--no-compare`	Skip local/remote consistency check	`False`

Examples#

# Remove single file
mc rm file.txt

# Remove empty directory
mc rm empty_dir/

# Remove directory and contents
mc rm -r directory/

# Remove multiple items
mc rm file1.txt file2.txt

# Remove from remote only
mc rm --remote-only remote_file.txt

# Force remove without comparison
mc rm --no-compare different_file.txt

Warning

Directory removal requires -r flag
--no-compare bypasses safety checks
Actions cannot be undone
Recursive removal deletes ALL contents

Safety Features#

Directory Protection * Must explicitly use -r for directories * Prevents accidental recursive deletion
Consistency Check * Verifies local/remote file matching * Can be bypassed with --no-compare
Scope Control * --remote-only for remote-only operations * Default affects both local and remote

Tip

Always verify paths before using -r
Use mc ls to confirm before large deletions
Consider --remote-only for selective cleanup

Using The CLI

Contents

Using The CLI#

Introduction#

Installation#

Adding Your API Key#

Configuration Steps#

Project Setup#

Creating a New Project#

Using an Existing Project#

File Upload/Download#

Basic Commands#

Example Directory Structures#

Common Usage Examples#

Uploading Files with Globus#

Upload Process Overview#

1. Create Upload Directory#

2. Transfer Files#

3. Finish Upload#

Managing Upload Directories#

Downloading Files with Globus#

Download Process#

File Operations#

Available Commands#

Listing Files (mc ls)#

Quick Reference#

Example Output#

Understanding the Output#

Making Directories (mc mkdir)#

Quick Reference#

Description#

Options#

Examples#

Moving Files/Directories (mc mv)#

Quick Reference#

Description#

Options#

Examples#

Deleting Files/Directories (mc rm)#

Quick Reference#

Description#

Options#

Examples#

Safety Features#