-2

I have a powerful server (32 cores, 128 GB RAM, MD Ryzen 9 7950X3D), and I installed Nginx.

I start load-testing it on a simple Health check using ab.

I notice that when I test it like this: ab -c 1000 -n 50000 -k "https://***.com/TestHealth"

I can reach 50k RPS, but when I test it like this:

ab -c 1000 -n 50000 "https://***.com/TestHealth"

I get only 2348 RPS.

The testing machine is a separate machine, the strongest I found (CPU-wise) on AWS.

Now I know that -k keeps keepAlive and this for sure helps with performance, but since the server was not heavily utilized on this load, I guess I can get much better results from it.

What are the best SSL and server settings I can do for this server? is there an organized process I can go through to make sure my server is optimized for the best possible performance for his HW and BW?

This is the results of vmstat 1 3 before the test:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 111926384 373764 14628824    0    0     1     2    3    4  0  0 100  0  0
 0  0      0 111926384 373764 14628832    0    0     0     0  511  874  0  0 100  0  0
 0  0      0 111926384 373764 14628832    0    0     0     0  638 1072  0  0 100  0  0

This is the results during the test (without -k):

vmstat 1 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 111884320 373852 14631856    0    0     1     2    3    4  0  0 100  0  0
 1  0      0 111882256 373852 14631996    0    0     0     0 35435 10614  2  1 97  0  0
 0  0      0 111882336 373860 14632032    0    0     0    38 38040 11449  2  2 96  0  0

This is the results during the test with -k:

vmstat 1 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 111862632 374048 14696080    0    0     1     2    4    4  0  0 100  0  0
 2  0      0 111860968 374048 14699616    0    0     0     0 100300 66310  1  3 96  0  0
 0  0      0 111859408 374048 14703228    0    0     0     0 100014 66075  1  3 95  0  0

Changin value worker_processes to 24: During -k test:

vmstat 1 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 112021120 374312 14761548    0    0     1     2    4    4  0  0 100  0  0
 2  0      0 112023288 374312 14765152    0    0     0     0 102319 66892  1  2 97  0  0
 0  0      0 112022048 374316 14768552    0    0     0    29 103237 67227  1  2 97  0  0

without:

vmstat 1 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 111983120 374328 14781508    0    0     1     2    4    4  0  0 100  0  0
 6  0      0 111982048 374328 14781644    0    0     0     0 36247 10694  2  1 97  0  0
 1  0      0 111980312 374328 14781788    0    0     0     0 34453 10596  1  1 97  0  0
0xPwn
  • 107
  • No, I don't look for planning the capacity, I'm looking for a manual/ process to optimize (tune) capacity on a specific server (Nginx, https) (server that hosts static files). – 0xPwn Jan 08 '24 at 11:37
  • I followed this: https://www.digitalocean.com/community/tutorials/how-to-optimize-nginx-configuration and similar, but still I can't go to 3K RPS on ab (the nginx just return 200, the CPU is never reached more than 80%, the ram is barely used, the BW is 1% used) – 0xPwn Jan 08 '24 at 11:39
  • What is the result of command vmstat 1 3? (when the test is running) – Romeo Ninov Jan 08 '24 at 11:49
  • @RomeoNinov vmstat 1 3 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 111926384 373764 14628824 0 0 1 2 3 4 0 0 100 0 0 0 0 0 111926384 373764 14628832 0 0 0 0 511 874 0 0 100 0 0 0 0 0 111926384 373764 14628832 0 0 0 0 638 1072 0 0 100 0 0 – 0xPwn Jan 08 '24 at 11:52
  • And run vmstat when you have the test run! – Romeo Ninov Jan 08 '24 at 11:53
  • 1
    @RomeoNinov vmstat 1 3 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 111884320 373852 14631856 0 0 1 2 3 4 0 0 100 0 0 1 0 0 111882256 373852 14631996 0 0 0 0 35435 10614 2 1 97 0 0 0 0 0 111882336 373860 14632032 0 0 0 38 38040 11449 2 2 96 0 0 – 0xPwn Jan 08 '24 at 11:57
  • 2
    Performance tuning is a black art and science. There is no objective best setting, as your real world load will be specific to you. The magic recipe for the best performance is to add detailed monitoring (not only of the process you're tuning but the also kernel / driver / IO / NIC / CPU load etc, etc. I like tools that generate graphs. Then the black art, make an educated guess as to what is the most critical limiter. Try to adjust the tuning of that limiter, evaluate the performance of the change, and then either keep or revert. Then focus on the next setting and/or bottleneck. – HBruijn Jan 08 '24 at 11:59
  • @0xPwn, is this result with -k or w/o? – Romeo Ninov Jan 08 '24 at 12:02
  • @RomeoNinov without the -k, with the -k I get ~50k RPS – 0xPwn Jan 08 '24 at 12:03
  • @0xPwn, run it with -k and add result of the same vmstat command please – Romeo Ninov Jan 08 '24 at 12:04
  • 1
    @RomeoNinov vmstat 1 3 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 0 111862632 374048 14696080 0 0 1 2 4 4 0 0 100 0 0 2 0 0 111860968 374048 14699616 0 0 0 0 100300 66310 1 3 96 0 0 0 0 0 111859408 374048 14703228 0 0 0 0 100014 66075 1 3 95 0 0 – 0xPwn Jan 08 '24 at 12:06
  • Please add in the location something like this: location / { root /TestHealth; aio threads; } and test again – Romeo Ninov Jan 08 '24 at 12:14
  • And set worker_processes to 24 for example – Romeo Ninov Jan 08 '24 at 12:15
  • @RomeoNinov This is my current settings: location /TestHealth { return 200 "hello"; } , worker_processes auto; – 0xPwn Jan 08 '24 at 12:17
  • 1
    The process is: Change configuration value, benchmark, repeat. – Gerald Schneider Jan 08 '24 at 12:18
  • @RomeoNinov , with it set to 24, this is the results without the -k: vmstat 1 3 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 111983120 374328 14781508 0 0 1 2 4 4 0 0 100 0 0 6 0 0 111982048 374328 14781644 0 0 0 0 36247 10694 2 1 97 0 0 1 0 0 111980312 374328 14781788 0 0 0 0 34453 10596 1 1 97 0 0 – 0xPwn Jan 08 '24 at 12:23

1 Answers1

0

As mentioned by Gerald Schneider the wise way is to change one parameter then test, analyse, repeat. Let me point you to some changes you can try:

  • Change/add worker_processes to be fixed value, less than number of cores (to give some place for the OS). Not use if your processor support something like hyperthreads, if yes, set worker_processes to be total number of hyperthreads - 4 (for example).
  • add in location aio threads

`

location /TestHealth  
    {      
        return 200 "hello";     
       aio threads;  
       }

`

Romeo Ninov
  • 5,932
  • 4
  • 20
  • 26