87

In the company I am working now there is a legacy service and its init script is using old SysvInit, but is running over systemd (CentOS 7).

Because there's a lot of computation, this service takes around 70 seconds to finish. I didn't configure any timeout for systemd, and didn't change the default configs at /etc/systemd/system.conf, but still when I execute service SERVICE stop my service is timing out after 60 seconds.

Checking with journalctl -b -u SERVICE.service I find this log:

Sep 02 11:27:46 service.hostname systemd[1]: Stopping LSB: Start/Stop
Sep 02 11:28:46 service.hostname SERVICE[24151]: Stopping service: Error code: 255
Sep 02 11:28:46 service.hostname SERVICE[24151]: [FAILED]

I already tried changing the DefaultTimeoutStopSec property at /etc/systemd/system.conf to 90s, but the timeout still happens.

Does anyone have any idea why is it timeouting at 60s? Is there somewhere else that this timeout value is configured? Is there a way I can check it?

This service runs with java 7 and to daemonize it, it uses JSVC. I configured the -wait parameter with the value 120.

MatthewRock
  • 6,986
Caesar Ralf
  • 1,103

4 Answers4

131

My systemd service kept timing out because of how long it would take to boot up also, so this fixed it for me:

  1. Edit your systemd file:

    • For modern versions of systemd: Run systemctl edit --full node.service (replace "node" with your service name).
      • This will create a system file at /etc/systemd/system/node.service.d/ that will override the system file at /usr/lib/systemd/system/node.service. This is the proper way to configure your system files. More information about how to use systemctl edit is here.
    • Directly editing system file: The system file for me is at /usr/lib/systemd/system/node.service. Replace "node" with your application name. However, it is not safe to directly edit files in /usr/lib/systemd/ (See comments)

  2. Use TimeoutStartSec, TimeoutStopSec or TimeoutSec (more info here) to specify how long the timeout should be for starting & stopping the process. Afterwards, this is how my systemd file looked:

    [Unit]
Description=MyProject
Documentation=man:node(1)
After=rc-local.service

    [Service]
WorkingDirectory=/home/myproject/GUIServer/Server/
Environment="NODE_PATH=/usr/lib/node_modules"
ExecStart=-/usr/bin/node Index.js
Type=simple
Restart=always
KillMode=process
TimeoutSec=900
    

    [Install]
WantedBy=multi-user.target
    • You can also view the current Timeout status by running any of these (but you'll need to edit your service to make changes! See step 1). Confusingly, the associated properties have a "U" in their name for microseconds. See this Github issue for more information:
      • systemctl show node.service -p TimeoutStartUSec
      • systemctl show node.service -p TimeoutStopUSec
      • systemctl show node.service -p TimeoutUSec

  3. Next you'll need to reload the systemd with systemctl reload node.service

  4. Now try to start your service with systemctl start node.service

  5. If that didn't work, try to reboot systemctl with systemctl reboot

  6. If that didn't work, try using the --no-block option for systemctl like so: systemctl --no-block start node.service. This option is described here: "Do not synchronously wait for the requested operation to finish. If this is not specified, the job will be verified, enqueued and systemctl will wait until the unit's start-up is completed. By passing this argument, it is only verified and enqueued."

    • There is also the option to use systemctl mask instead of systemctl start. For more info see here.

Updates from Comments:

  • TimeoutSec=infinity: Instead of using "infinity" here, put a large amount of time instead, like TimeoutSec=900 (15 min). If the application takes "forever" to exit, then it's possible that it will block a reboot indefinitely. Credit @Alexis Wilke and @JCCyC
  • Instead of editing /usr/lib/systemd/system, try systemctl edit instead or edit /etc/systemd/system to override them instead. You should never edit service files in /usr/lib/. Credit @ryeager and @0xC0000022L

** Update from systemd source docs ** When specified "infinity" as a value to any of these timeout params, the timeout logic is disabled.

JobTimeoutSec=, JobRunningTimeoutSec=,TimeoutStartSec=,  TimeoutAbortSec=

The default is "infinity" (job timeouts disabled), except for device units where JobRunningTimeoutSec= defaults to DefaultTimeoutStartSec=.

Reference: enter link description here

Similarly this logic applies to service level and laid out clearly in URL below. Reference: enter link description here

imranxahmed
  • 3
  • 1
Katie
  • 1,562
  • 13
    TimeoutSec=infinity — wouldn't it be possible that this block a reboot indefinitely? What if it takes "forever" for that process to exit? I'd suggest a large amount, like 5min, but probably not infinity... – Alexis Wilke Dec 13 '16 at 06:09
  • 6
    you shouldn't be editing service files in /usr/lib, you should edit them or override them in /etc/systemd/system – ryeager Jan 08 '18 at 19:17
  • 5
    While the gist of the advice is sound, I have to concur with @ryeager ... modern versions of systemd offer systemctl edit (and mask to disable them with brute force, as opposed to disable) for that very purpose. You should never edit the files in /usr/lib/systemd. – 0xC0000022L May 15 '18 at 11:43
  • 3
    TimeoutSec=infinity didn't work here, I used TimeOutSec=900 (15 min) and that saved my posterior. -- I needed to run systemctl daemon-reload afterwards, before restarting the service. – JCCyC Nov 01 '18 at 15:05
  • 1
    It works, add TimeoutSec=900 – Stony Jan 06 '20 at 06:02
13

Running systemctl show SERVICE_NAME.service -p TimeoutStopUSec I could at least see the timeout set by systemd to my service.

I changed the script to a regular unit file one in order for it work properly.

Caesar Ralf
  • 1,103
5

Instead of editing the package's service file in /usr/lib/systemd/system/ which will get overridden on package upgrade, use:

sudo EDITOR=/bin/vi systemctl edit <service-name>

(Yes, nano is the default editor! Eeugh!)

This will safely edit the file /etc/systemd/system/service-name.service.d/override.conf.

This file need only contain:

[Service]
# Override default 90 second timeout in pathological conditions
TimeoutStopSec=5  # or whatever value you want

See the overall effect by typing:

systemctl cat <service-name>
Tom Hale
  • 30,455
  • 1
    The question states, in two ways, that the package did not have a service unit file, and was relying upon an autogenerated one. The questioner notes in another answer that xe simply wrote a service unit xyrself, back in 2015. Compare https://unix.stackexchange.com/a/560998/5132 . – JdeBP Jan 25 '20 at 11:07
  • JdeBP: OP wrote later that he'd changed it to use a unit file. But even if he hadn't, the question title doesn't include that detail, and folks (like myself) get sent here from Google. Although this answer has now been subsumed by Katie's, it was still useful to me because it reiterated something that's easy to miss and because it very succinctly answers the question (as titled -- though, granted, not as fully described). – johncip Nov 30 '20 at 23:04
0

To achieve what you have wanted I use following 3 commands,

To check the default Timeout value,

systemctl show yourServiceName.service | grep ^Timeout

To increase default Timeout value,

mkdir /etc/systemd/system/yourServiceName.service.d

echo -e "[Service]\nTimeoutStartSec=180" | tee /etc/systemd/system/yourServiceName.service.d/startup-timeout.conf

Where,

  • yourServiceName : is the service name. E.g, fluentd, weblogic, etc.
  • TimeoutStartSec=180 : Here, I have use 180sec/3min as timeout value.
arif
  • 1,459