GLC Networks

Linux, Mikrotik, Ubiquity, Radius manager

Solving scalability issue on apache with MPM event parameters

this article is about Solving scalability issue on apache with MPM event parameters. So today, we got an issue from users that connects to our webserver.

Background:

  • Total user is around 400-500.
  • We host an elearning application, and in an event (for example exam) and users are accessing our server simultaneously
  • the application is a web-based and users accessing it either from their laptop or mobile devices
  • we used the default configuration of apache webserver (apache mpm_event)

Symptoms:

  • Some users reported can access the website smoothly but many are not
  • But sometimes, the user that can open the page previously, become unable to view the page.
  • We have check bandwidth, CPU usage, RAM usage, of webserver and database server, and they are normal. even though the web server is accessed by many users concurrently, server resources usage are relatively low.

Analyses (Solving scalability issue on apache with MPM event parameters):

  • from the webserver log we found that there were many messages related to “reached MaxRequestWorkers setting”, therefore it seems like the issue is related to our default apache setting
  • based on apache documentation,MaxRequestWorkers defines the maximum number of user that can connect to our web server simultaneously
  • on our default apache configuration we found that this number is quite low: “MaxRequestWorkers 150” compared to total simultaneous users that accessing the server.
  • therefore we need to increase the value in order to serve more users
  • note: the parameter (directive) above is linked with other directives as well :-(. thus, you need to read the documentation about other parameters as well. see the table below

When you use apache with mpm event module, here are several important parameters that you need to know to Solving scalability issue on apache with MPM event parameters.

directive explanation
StartServers: Number of child server processes created at startup The StartServers directive sets the number of child server processes created on startup. As the number of processes is dynamically controlled depending on the load, (see MinSpareThreads, MaxSpareThreads, MinSpareServers, MaxSpareServers) there is usually little reason to adjust this parameter.
The default value differs from MPM to MPM. worker and event default to StartServers 3; prefork defaults to 5; mpmt_os2 defaults to 2.
ServerLimit: Upper limit on configurable number of processes For the prefork MPM, this directive sets the maximum configured value for MaxRequestWorkers for the lifetime of the Apache httpd process. For the worker and event MPMs, this directive in combination with ThreadLimit sets the maximum configured value for MaxRequestWorkers for the lifetime of the Apache httpd process. For the event MPM, this directive also defines how many old server processes may keep running and finish processing open connections. Any attempts to change this directive during a restart will be ignored, but MaxRequestWorkers can be modified during a restart.
Special care must be taken when using this directive. If ServerLimit is set to a value much higher than necessary, extra, unused shared memory will be allocated. If both ServerLimit and MaxRequestWorkers are set to values higher than the system can handle, Apache httpd may not start or the system may become unstable.
With the prefork MPM, use this directive only if you need to set MaxRequestWorkers higher than 256 (default). Do not set the value of this directive any higher than what you might want to set MaxRequestWorkers to.
With worker, use this directive only if your MaxRequestWorkers and ThreadsPerChild settings require more than 16 server processes (default). Do not set the value of this directive any higher than the number of server processes required by what you may want for MaxRequestWorkers and ThreadsPerChild.
With event, increase this directive if the process number defined by your MaxRequestWorkers and ThreadsPerChild settings, plus the number of gracefully shutting down processes, is more than 16 server processes (default).
Note
There is a hard limit of ServerLimit 20000 compiled into the server (for the prefork MPM 200000). This is intended to avoid nasty effects caused by typos. To increase it even further past this limit, you will need to modify the value of MAX_SERVER_LIMIT in the mpm source file and rebuild the server.
MaxRequestWorkers: Maximum number of connections that will be processed simultaneously The MaxRequestWorkers directive sets the limit on the number of simultaneous requests that will be served. Any connection attempts over the MaxRequestWorkers limit will normally be queued, up to a number based on the ListenBacklog directive. Once a child process is freed at the end of a different request, the connection will then be serviced.
For non-threaded servers (i.e., prefork), MaxRequestWorkers translates into the maximum number of child processes that will be launched to serve requests. The default value is 256; to increase it, you must also raise ServerLimit.
For threaded and hybrid servers (e.g. event or worker) MaxRequestWorkers restricts the total number of threads that will be available to serve clients. For hybrid MPMs the default value is 16 (ServerLimit) multiplied by the value of 25 (ThreadsPerChild). Therefore, to increase MaxRequestWorkers to a value that requires more than 16 processes, you must also raise ServerLimit.
MaxRequestWorkers was called MaxClients before version 2.3.13. The old name is still supported.
ThreadsPerChild: Number of threads created by each child process This directive sets the number of threads created by each child process. The child creates these threads at startup and never creates more. If using an MPM like mpm_winnt, where there is only one child process, this number should be high enough to handle the entire load of the server. If using an MPM like worker, where there are multiple child processes, the total number of threads should be high enough to handle the common load on the server.
ThreadLimit: Sets the upper limit on the configurable number of threads per child process This directive sets the maximum configured value for ThreadsPerChild for the lifetime of the Apache httpd process. Any attempts to change this directive during a restart will be ignored, but ThreadsPerChild can be modified during a restart up to the value of this directive.
Special care must be taken when using this directive. If ThreadLimit is set to a value much higher than ThreadsPerChild, extra unused shared memory will be allocated. If both ThreadLimit and ThreadsPerChild are set to values higher than the system can handle, Apache httpd may not start or the system may become unstable. Do not set the value of this directive any higher than your greatest predicted setting of ThreadsPerChild for the current run of Apache httpd.
The default value for ThreadLimit is 1920 when used with mpm_winnt and 64 when used with the others.
There is a hard limit of ThreadLimit 20000 (or ThreadLimit 100000 with event, ThreadLimit 15000 with mpm_winnt) compiled into the server. This is intended to avoid nasty effects caused by typos. To increase it even further past this limit, you will need to modify the value of MAX_THREAD_LIMIT in the mpm source file and rebuild the server.
MaxConnectionsPerChild: Limit on the number of connections that an individual child server will handle during its life The MaxConnectionsPerChild directive sets the limit on the number of connections that an individual child server process will handle. After MaxConnectionsPerChild connections, the child process will die. If MaxConnectionsPerChild is 0, then the process will never expire.

Setting MaxConnectionsPerChild to a non-zero value limits the amount of memory that process can consume by (accidental) memory leakage.

MinSpareThreads: Minimum number of idle threads available to handle request spikes Minimum number of idle threads to handle request spikes. Different MPMs deal with this directive differently.
worker and event use a default of MinSpareThreads 75 and deal with idle threads on a server-wide basis. If there aren’t enough idle threads in the server then child processes are created until the number of idle threads is greater than number. Additional processes/threads might be created if ListenCoresBucketsRatio is enabled.
mpm_netware uses a default of MinSpareThreads 10 and, since it is a single-process MPM, tracks this on a server-wide bases.
mpmt_os2 works similar to mpm_netware. For mpmt_os2 the default value is 5.
MaxSpareThreads: Maximum number of idle threads Maximum number of idle threads. Different MPMs deal with this directive differently.
For worker and event, the default is MaxSpareThreads 250. These MPMs deal with idle threads on a server-wide basis. If there are too many idle threads in the server then child processes are killed until the number of idle threads is less than this number. Additional processes/threads might be created if ListenCoresBucketsRatio is enabled.
For mpm_netware the default is MaxSpareThreads 100. Since this MPM runs a single-process, the spare thread count is also server-wide.
mpmt_os2 works similar to mpm_netware. For mpmt_os2 the default value is 10.
Restrictions
The range of the MaxSpareThreads value is restricted. Apache httpd will correct the given value automatically according to the following rules:
mpm_netware wants the value to be greater than MinSpareThreads.
For worker and event, the value must be greater or equal to the sum of MinSpareThreads and ThreadsPerChild.
   

thank you for reading Solving scalability issue on apache with MPM event parameters

Leave a Reply

Your email address will not be published. Required fields are marked *

GLC Networks © 2016 Frontier Theme