1

I'm losing files in /tmp on current Ubuntu (22.04.2), possibly related to virtualisation. Any insights appreciated. This is not a normal cleanup; the file is lost immediately.

I have two systems: a remote VPS, and a local staging system, both running the same software. On both systems, /tmp is not a tmpfs; it's part of the LVM volume group mounted at /.

Apache runs program1 and program2 immediately after each other:

  1. program1 creates a new subdirectory under /tmp, with 2 files: file1 (a Python source file) and file2 (the output of file1). If debug is enabled, file1 is retained; otherwise, it it deleted after creating file2 (with a C library remove)
  2. program2 returns file2 to the client

This all works as expected on the staging system. If debug is enabled, the new /tmp/xxx directory is still on the filesystem, and contains both file1 and file2. If debug isn't enabled, it contains only file2.

On the VPS:

  1. if debug is not enabled, everything works, but there's nothing left under /tmp
  2. if debug is enabled, program2 fails, reporting that file2 doesn't exist

I have 'fixed' the code by removing the C lib remove, so that file1 is always retained, but I don't understand what's going on - it appears that the file remove has somewhere triggered a remove of the entire directory.

EML
  • 470
  • Do you have functionality in the application that removes these files after certain times? I'd suspect permissions and on one filesystem you application doesn't have delete privileges and on the other filesystem it does. If you are auditing the filesystem, OS logs should show what process is deleting the file and when. – DubStep Jun 01 '23 at 13:30
  • @DubStep - no, nothing in the app deletes the files, apart from this one remove op. The app just relies on the OS to eventually clean up everything else. The app runs as www-data on both systems, and both /tmp are drwxrwxrwt, as expected. No auditing enabled. – EML Jun 01 '23 at 13:37
  • 1
    /tmp traditionally has been a shared space for all local services and users and also been a big source of security problems for all kinds of services. Symlink attacks and DoS vulnerabilities due to guessable /tmp temporary files are common. That's why systemd introduced the PrivateTmp=yes setting. If true that sets up a new file system namespace for the executed processes and mounts private /tmp/ and /var/tmp/ directories inside it that are not shared by processes outside of the namespace. – HBruijn Jun 01 '23 at 13:38
  • 1
    Afaik PrivateTmp=yes is the default for Apache2 in the recent Ubuntu versions, which may explain some of your problems. – HBruijn Jun 01 '23 at 13:38
  • @HBruijn - interesting - I think you must be right, but both machines have PrivateTmp true. So I presume that one Apache managed to run both program1 and program2 in the same process, and the other has them in different processes, so maybe the remove is irrelevant. Both apaches have mod_cgid. I think the answer must be to set PrivateTmp off. But why is it on anyway? This isn't a system service - why would systemd care if I want to have two Apache processes talking to each other? – EML Jun 01 '23 at 14:39
  • It might be interesting to see what happens when you don't use /tmp to store your files and/or (temporarily) disable the PrivateTmp=true setting by changing the apache2.service systemd unit file (see https://serverfault.com/a/840999/37681) – HBruijn Jun 01 '23 at 14:52

0 Answers0