xvda1 is 100% full, What is it? how to fix?

Question

I'm running a Linux instance on EC2 (I have MongoDB and node.js installed) and I'm getting this error:

Cannot write: No space left on device

I think I've tracked it down to this file, here is the df output

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvda1             1032088   1032088         0 100% /

The problem is, I don't know what this file is and I also don't know if this file is even the problem.

So my question is: How do I fix the "No space left on device" error?

score 102 · Accepted Answer · answered Nov 13 '11 at 03:43

102

That file, / is your root directory. If it's the only filesystem you see in df, then it's everything. You have a 1GB filesystem and it's 100% full. You can start to figure out how it's used like this:

sudo du -x / | sort -n | tail -40

You can then replace / with the paths that are taking up the most space. (They'll be at the end, thanks to the sort. The command may take awhile.)

answered Nov 13 '11 at 03:43

David Schwartz

31,609
2
56
85

31

To get the output in a human-readable format, you can use sudo du -x -h / | sort -h | tail -40 (from this answer). – mkobit Jun 21 '16 at 14:09
3

For those on micro AWS AMI instances, this can take a minute or so to run. Be patient! – Dr Rob Lang Nov 15 '18 at 11:31
what to do with this:sort: write failed: /tmp/sortGmL8oF: No space left on device – dOM Dec 03 '18 at 09:28
1

@dOM Ouch. Try to clean off some space on /tmp. Or, if you must, narrow it down step by step with commands like du -xhs /*. – David Schwartz Dec 03 '18 at 09:45
du -x -h / | sort -h | tail -40 | sort -h -r can be used to sort in descending order when using human-readable output. – Vigs May 11 '19 at 17:53

score 28 · Answer 2 · answered Sep 22 '17 at 13:49

I know i am replying in this thread after nearly 5 years but it might help someone, I had the same problem, i had m4.xlarge instance df -h told that the /dev/xvda1 was full, - 100%

Filesystem      Size  Used Avail Use% Mounted on
udev            7.9G     0  7.9G   0% /dev
tmpfs           1.6G  177M  1.4G  12% /run
/dev/xvda1      7.7G  7.7G     0 100% /
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
tmpfs           1.6G     0  1.6G   0% /run/user/1000

i tried to solve it here are the steps

sudo find / -type f -printf '%12s %p\n' 2>/dev/null|awk '{if($1>999999999)print $0;}'

Helped me to know that it was the docker container that was talking all my space so i push all my container to my docker registry then did sudo rm -rf /var/lib/docker/ it cleared up my space :) hope it helps someone :)

Thanks. How đi you push your container to your docker resgistry? It is not good just to delete the xxx-json.log file right? — Freelensia, Jun 23 '21 at 13:45

score 11 · Answer 3 · answered Nov 13 '11 at 05:15

If you are running an EBS boot instance (recommended) then you can increase the size of the root (/) volume using the procedure I describe in this article:

Resizing the Root Disk on a Running EBS Boot EC2 Instance
http://alestic.com/2010/02/ec2-resize-running-ebs-root

If you are running an instance-store instance (not recommended) then you cannot change the size of the root disk. You either have to delete files or move files to ephemeral storage (e.g., /mnt) or attach EBS volumes and move files there.

Here's an article I wrote that describes how to move a MySQL database from the root disk to an EBS volume:

Running MySQL on Amazon EC2 with EBS
http://aws.amazon.com/articles/1663

...and consider moving to EBS boot instances. There are many reasons why you'll thank yourself later.

I'm running on EBS, it's fairly cheap to expand the root disc right? Fortunately I don't have to deal with MySQL, My projects are currently Mongo/Redis.
some great material here. +1 — , Nov 13 '11 at 06:31

score 4 · Answer 4 · answered May 04 '20 at 17:41

Paulo was on the right track for me, but when I tried to run

sudo apt autoremove

it responded:

Reading package lists... Error!
E: Write error - write (28: No space left on device)
E: IO Error saving source cache
E: The package lists or status file could not be parsed or opened.

First, I had to run

sudo apt-get clean

That cleared just enough space for me to run 'sudo apt autoremove', and that took me from 100% full on /dev/xvda1 to 28%.

score 4 · Answer 5 · edited Dec 04 '23 at 20:26

4

I have just solved that problem by running this command:

sudo apt autoremove

A lot of old packages were removed, freeing up 5 gigabytes, for instance there were many packages like linux-aws-headers-4.4.0-1028

edited Dec 04 '23 at 20:26

CPlus

103

answered Dec 27 '18 at 22:50

Paulo

41

score 4 · Answer 6 · answered Jun 24 '19 at 17:36

I've recently run into this issue on Amazon Linux. My crontab outbound email queue /var/spool/clientmqueue was 4.5GB.

I solved it by:

Locating large files: sudo find / -type f -size +10M -exec ls -lh {} \;
Deleting large files: /bin/rm -f <path-to-large-file>
Restart server instance

Problem solved!

score 1 · Answer 7 · answered Oct 02 '20 at 17:01

Hope this helps to those who are using codedeploy agent and having a similar issue.

I was using Amazon Linux EC2 instance and my directory was 100% full. first, to run the command I deleted all files /var/log/journal/.

then run this command. sudo du -xhc / and found that out of 8GB, codedeploy-agent/deployment-root folder was using 5.1GB space.

By default codedeploy-agent store last 5 archive so i changed :max_revision from 5 to 2 in /etc/codedeploy-agent/conf/codedeployagent.yml

score 1 · Answer 8 · edited Mar 18 '21 at 13:11

1

Use du -hs * | sort -rh | head -5 to check top 5 usage, then rm -rf name to remove junk like large size log file or archived within logs folder

edited Mar 18 '21 at 13:11

Andrew Schulman

8,916

answered Mar 18 '21 at 12:47

Nilesh Kumar

11

score 1 · Answer 9 · answered Jun 26 '19 at 09:50

1

It could be coming from Jenkins or Docker. To solve that, you should clean Jenkings logs and set it's size.

answered Jun 26 '19 at 09:50

T.Todua

214

score 0 · Answer 10 · answered Oct 21 '22 at 05:43

If we really have used up the current space, we'll need to extend the virtual disk (EBS) in EC2.

After you increase the EBS volume for EC2 from either the web or CLI, follow this AWS article to finish expanding the disk from the OS level.

In short, only two commands are needed if the virtual technology in your EC2 is Xen. No need to unmount/mount the dev or restart the instance as previous posts suggested.

sudo growpart /dev/xvda 1
sudo xfs_growfs -d /

xvda1 is 100% full, What is it? how to fix?

10 Answers10

Linked