Apache - No space left on device / Failed to create proxy Mutex
A Big Thanks to "https://help.directadmin.com/item.php?id=110”
Restarting Apache in one of the Servers failed with "No Space left on the Device" Error and we have enough space in all available partitions.
This is something to deal with Server Kernel semaphores.
On Linux, A semaphore is a System V IPC object that is used to control utilization of a particular process. Refer https://www.tldp.org/LDP/tlk/ipc/ipc.html
Semaphores are a shareable resource that take on a non-negative integer value. They are manipulated by the P (wait) and V (signal) functions, which decrement and increment the semaphore, respectively. When a process needs a resource, a “wait” is issued and the semaphore is decremented. When the semaphore contains a value of zero, the resources are not available and the calling process spins or blocks (as appropriate) until resources are available. When a process releases a resource controlled by a semaphore, it increments the semaphore and the waiting processes are notified.”
The system does n't have actual resource available to serve the request, so either we need to configure the Kernel Semaphores or clear the Old entries to get it back.
kernel.sem = SEMMSL SEMMNS SEMOPM SEMMNI
SEMMSL maximum number of semaphores per array
SEMMNS maximum semaphores system-wide
SEMOPM maximum operations per semop call
SEMMNI maximum arrays
[[email protected] ~]# cat /etc/sysctl.conf |grep kernel.sem
kernel.sem = 256 32000 100 142
[[email protected] ~]#[[email protected] ~]# cat /proc/sys/kernel/sem
256 32000 100 142
[[email protected] ~]#
[[email protected] ~]# ipcs -l | awk ‘FNR>=7 && FNR<=15’
—— Semaphore Limits ——–
max number of arrays = 142
max semaphores per array = 256
max semaphores system wide = 32000
max ops per semop call = 100
semaphore max value = 32767—— Messages: Limits ——–
[[email protected] ~]#[[email protected] ~]# ipcs -ls
—— Semaphore Limits ——–
max number of arrays = 142
max semaphores per array = 256
max semaphores system wide = 32000
max ops per semop call = 100
semaphore max value = 32767
To Clear , we can execute the following
ipcs | grep apache | awk ‘{print $2}’ > sem.txt
for i in `cat sem.txt`; do { ipcrm -s $i; }; done;
| For Automating the removal of apache semaphores with ipcs/ipcrm, Please refer https://help.directadmin.com/item.php?id=572 |
最终，参考这篇文章，我的解决方案是将/etc/sysctl.conf文件中sem的数值改大，然后执行/sbin/sysctl -p搞定。