Upload From Pc to Supercomputer Command Line

Filesystems and Storage

Overview

Teaching: xv min
Exercises: ten min

Questions

• Where tin can I shop my data?

• What is the difference between scratch and dwelling house filestore?

Objectives

• Understand the divergence betwixt abode and scratch directories

• Sympathize how to re-create files between your computer and your Supercomputing Wales home/scratch directories

Filesystems and Storage

What is a filesystem?

Storage on almost compute systems is not what and where you think they are! Physical disks are arranged together into a virtual volume; this virtual volume may represent one filesystem, or may exist divided upwards, or partitioned, into multiple filesystems. And your directories then reside within one of these fileystems. Filesystems are accessed over the network through mount points.

There are multiple storage/filesystems options available for yous to do your work. The most common are:

• abode: where y'all land when you starting time login. 50 GB per user. Slower admission, backed up (Cardiff only). Used to store your piece of work long term.
• projection: shared between all users of a project. Same filesystem as home.
• scratch: temporary working space. Faster admission, not backed up. Larger quota, just one-time files might become deleted. DON'T STORE RESULTS HERE!

Here's a synopsis of filesystems on Hawk in Cardiff:

Name	Path	Default Quota	Disk Size	Backed Up
Home	/home/user.name	50GB	420TB	Yeah
Project	/dwelling/scwXXXX	Negotiable	12TB (same disk as domicile)	Yes
Scratch	/scratch/user.name	20TB + 10million files	1200TB	No

and on Sunbird in Swansea:

Name	Path	Default Quota	Disk Size	Backed Upwards
Dwelling	/home/user.proper name	100GB	231TB	No
Projection	/home/scwXXXX	Negotiable	231TB (aforementioned deejay as domicile)	No
Scratch	/scratch/user.name	20TB + 10million files	808TB	No

Important!! Ensure that you don't store anything longer than necessary on scratch, this tin negatively affect other people'due south jobs on the system.

Accessing your filestore

How much quota do I have left on my home directory?

Login to a head node (e.g. sunbird.swansea.ac.great britain or hawklogin.cf.air conditioning.united kingdom of great britain and northern ireland) and run the myquota command. This will tell y'all how much infinite is left in your dwelling house directory.

            HOME DIRECTORY a.cos      Filesystem    used   quota   limit   grace   files   quota   limit   grace     /lustrehome   48.4G    100G    110G       -  206182  220000  230000       -  SCRATCH DIRECTORY a.cos      Filesystem    used   quota   limit   grace   files   quota   limit   grace        /scratch  63.29G  19.56T  xx.06T       -  313075  9851800 10101800       -                      

Group Filestore

If y'all have multiple collaborators working on a particular projection and would similar to share common software or data beyond the project, then it will be convenient for you to use a shared filestore. These tin can exist created on /home (for long-term use, due east.thousand. software) or on /scratch (for short-term data). If yous would like one setup for you, yous tin raise a support ticket or speak to ane of your local RSEs.

How much scratch take I used?

The df command tells y'all how much deejay space is left. The -h argument makes the output easier to read, it gives man readable units like Chiliad, G and T for Megabyte, Gigabyte and Terrabyte instead of but giving output in bytes. Past default df volition give us the complimentary space on all the drives on a arrangement, but we can merely enquire for the scratch drive past adding /scratch as an argument later on the -h.

            Filesystem                                Size  Used Avail Utilise% Mounted on 172.two.1.51@o2ib:172.ii.1.52@o2ib:/scratch  692T   57T  635T   nine% /scratch                      

Copying data from your PC to Supercomputing Wales

You lot can copy files to/from your Supercomputing Wales home and scratch drives using the secure copy protocol (SCP) or secure file transfer protocol (SFTP) and connecting to Sunbird or Hawk.

Copying data using Filezilla

Filezilla is a graphical SCP/SFTP customer available for Windows, Mac and Linux. You tin can download it from Filezilla download

Open filezilla and type sftp://sunbird.swansea.air conditioning.uk or sftp://hawklogin.cf.air conditioning.uk into the host box. Enter your username and password in the username/countersign boxes.

Click Quickconnect and a connectedness volition be started. The first time you connect you lot will be asked to verify the host central, tick the "Always trust this host, add together fundamental to the cache" box to terminate this message appearing again in hereafter.

Y'all should now take some files in the correct manus side of the window. These are on the remote system, the list on the left paw side is your local organisation.

Files can exist transferred either by dragging and dropping them from one side to the other. Or you can correct click on a remote file and cull "Download" or a local file and choose "Upload".

Y'all can change directory on the remote host by typing a path into the "Remote site:" box. For example type in /scratch/user.name (where user.proper noun is your username) to access your scratch directory.

Copying on the control line using SFTP or SCP

If y'all adopt to employ a command line interface to copy files then see the reference material on using the SFTP and SCP commands.

Exercises

Using the df command.

1. Login to a login node
2. Run the command df -h.
3. How much space does /scratch take left?
4. If you had to run a large job requiring 10TB of scratch space, would there be enough space for it?

Using the myquota control.

1. Login to a login node.
2. Run the myquota control.
3. How much space have you used and how much do you have left?
4. If yous had a job that resulted in 60GB of files would you accept enough space to store them?

Copying files.

1. Login to a login node.
2. Create a file chosen hello.txt by using the nano text editor (or the editor of your choice) and typing nano hello.txt. Enter some text into the file and press Ctrl+10 to save it.
3. Use either Filezilla or SCP/SFTP to copy the file to your computer.
4. Create a file on your estimator using a text editor. Copy that file to your Supercomputing Wales home directory using Filezilla or SCP/SFTP and examine its conents with nano on the login node.

Central Points

• The habitation directory is the default place to store data.

• The scratch directory is a larger space for temporary files.

• On Militarist in Cardiff home is backed up only is as well a slower disk.

• On Sunbird in Swansea neither is backed up.

• Quotas on home are much smaller than scratch.

hinkelnand1938.blogspot.com

Source: https://supercomputingwales.github.io/SCW-tutorial/03-moving-data/

Share this post