10

I have a huge log file I need to delete on a production web server. I'm worried it'll bring the system to a crawl if I rm it on Linux. Any brilliant ideas?

Update:

Filesystem: ext3

Partition: /var (mostly logs and MySQL data)

Log file is no longer being written to. (No additional data is being appended)

Web Server is LAMP (lots of IO)

objectoriented
  • 143
  • 1
    Duplicate of http://serverfault.com/questions/128012/how-to-make-rm-faster-on-ext3-linux ?

    What is your underlying file system?

     – Ben DeMott Dec 02 '11 at 05:55
  • ext3 - hoping for something more inspiring that ionice might work... I read the man page (http://linux.die.net/man/1/ionice) and it sounds like the right tool... I'd like a someone with experience using it to chime in. I don't feel very adventurous when it comes to production servers. – objectoriented Dec 02 '11 at 06:06
  • Did any of these solve the problem? – ewwhite Mar 05 '12 at 02:14
  • No, I've delayed dealing with the problem so far. I'll update this topic after I try one of the suggested methods. I'm leaning towards zeroing the file. – objectoriented Mar 06 '12 at 18:01

3 Answers3

12

It may be faster to zero/truncate the file than remove it. I also mention this because that's a really large log file, so there must be a tremendous amount of process activity writing to it. Try : > /path/to/logfile.log if you're not in a position to stop and start the production services.

ewwhite
  • 198,150
  • Thanks! I was able to stop the process writing to the file so no additional data is being appended to the file. – objectoriented Dec 02 '11 at 15:58
  • Go ahead and zero it. – ewwhite Dec 02 '11 at 16:25
  • Great idea, thank you! Took absolutely ZERO io when truncating a 2 GB file, which normally halts the server for 20 seconds. – Shane N Mar 23 '15 at 21:29
  • Truncating a file to zero bytes need to perform the same amount of work to free the blocks as deleting the file does. But truncating many times removing a few MB each time can spread the load over a long time if sleeps are inserted between each truncate. – kasperd Apr 28 '15 at 07:28
10

ionice -c3 rm yourfile.log is your best shot, then rm will belong to idle I/O class and only uses I/O when any other process does not need it. ext3 is not stellar when deleting huge files and there's not very much you can do about it. Yes, the rm command will slow down your system. The amount of slowness and the duration of the deletion is something one can only guess, it depends so much on hardware, kernel version and ext3 file system creation settings.

For log servers and other servers with large files I tend to use XFS, as it is very fast with them.

Janne Pikkarainen
  • 32,082
  • 1
    Thanks Janne. I worry there won't much idle time? I think ionice -c2 -n7 rm big.log is more appropriate. – objectoriented Dec 02 '11 at 06:17
  • 1
    I have no idea if your server has lots of free disk I/O time or not. But which one of you is more important to you: 1) time it takes to remove the file or 2) your server stays ~smooth during deletion? – Janne Pikkarainen Dec 02 '11 at 07:06
  • 1
    Method with least disruption is the goal. My point here is that thing is a web server (LAMP to be specific) so it's business is IO. I just don't see the disks sitting idle very much. I worry a giant file removal would take days or longer if the system is waiting for periods of zero IO. – objectoriented Dec 02 '11 at 15:53
  • IONICE is the kicker here. I spent hours trying to figure out what was slowing down my server while deleting a substantial amount of files.

    "find . -type f -delete -print" was using 99.9% of my I/O and other processes were stacking up because they could not complete their I/O.

     – Nick Woodhams Aug 15 '12 at 11:53
1

Alternate solution is having separate disks and cycle between them. So when your done logging to one disk, you swap to the other, and then you could use lots of IO to remove stuff, and not burden the active disk.

Hans-Henrik Stærfeldt
  • 11