My User Disk Space Quota in the lab has exceeded. How can I clean the unnecessary files?

The steps to check your current disk-space usage in your home directory in the lab and clean the unnecessary files are given below.

  1. Login into the lab console at either https://e.cloudxlab.com/ or https://f.cloudxlab.com/ using your lab userid and password. Detailed instructions to login are given here

  2. Run the following command to check the disk-space usage in your home directory:
    cd ~
    du -sch .[!.]* * |sort -h

  3. If the total size reported by the command above is less than 3G (or 3072M), you don’t need to do anything right now, and you can ignore the rest of the steps.
    If the size reported by the command above is more than 3G (or 3072M), then you need to delete some files in your home directory in order to reduce your disk usage below 3GB (or else you will not be able to create any new files in the home directory until your disk usage falls below 3GB).

  4. Before deleting any files, we recommend that you backup your important files offline to your own computer. You can upload and download files between your home directory and your local computer using the program WinSCP on Windows, or the program scp on Mac/Linux.
    Detailed instructions to upload and download files are given here

  5. If your current disk usage is above 3GB, then after backing up your important files to your local computer, you can delete unneeded files from your home directory to reduce your disk usage.

  6. To delete files in your home directory use:
    rm file-name

  7. To delete a directory and ALL of its contents including its subdirectories use:
    WARNING: Please be very careful while running this command, it will delete a directory and ALL of its contents including its subdirectories without asking for any confirmation!!!
    rm -r directory-name

  8. After cleaning out your un-needed files and directories, you can re-check if your disk usage is below 3GB, by again using this command:
    cd ~
    du -hs

1 Like

My local files are only around 500 MB.

All my disk space seems to be taken up by conda and pip folders. Things might break if delete them. (I did install or upgrade a few libraries based on the errors I got; but I really don’t recall what all? :sweat_smile:)

If there is a way to factory reset things, could you please do the same? It’s okay if I lose the files.

1 Like

Hey did ur problem got solved ,I am facing the same issue.

1 Like

Hi,
We know that conda environment takes up huge space. So, it would be great if you can help us understand generic packages which would be required by other users also and we would install them in our lab. Usually, .ivy2 and .local directories take up space so please delete them.

If you delete your environment and recreate the environment with the Python 3 it will take up less space. To use python3 run export PATH=/usr/local/anaconda/bin:$PATH . It is also given in the notes section of lab.

2 Likes

Hi ,

My “.local” folder is showing 1.5GB usage for me. Can I delete that folder or is it necessary to keep? And also “.cache” folder is using 509MB Can I delete that folder as well?

Regards,
Mahesh

You can try deleting them. These folders will again get recreated automatically if required.

1 Like

[abchandravansi8369@cxln4 ~]$ cd ~
[abchandravansi8369@cxln4 ~]$ du -hs
1.7G

As the Above steps my Space is Less Than 3GB . Then why i am facing issue with :
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/abchandravansi8369 is exceeded: quota = 4294967296 B = 4 GB but diskspace consumed = 4562096540 B = 4.25 GB0

This is the size of your Linux home directory.

While here you the error is corresponding to Hadoop file system. Try removing files from hdfs using: hadoop fs -rm commands.