Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
systems:linux_setup_and_configuration [2015/12/09 14:43] smayr [Cleaning Hacked Server] |
systems:linux_setup_and_configuration [2019/07/29 14:27] (current) ajdavis [Linux Setup <del>Outdated as of 2019</del>] |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | == Linux Setup == | + | == Linux Setup (Outdated as of 2019) == |
=== Installation === | === Installation === | ||
# Download CD images (ISO files) from http:// | # Download CD images (ISO files) from http:// | ||
Line 55: | Line 55: | ||
AuthUserFile / | AuthUserFile / | ||
<Limit GET> | <Limit GET> | ||
+ | # Require an authenticated user | ||
Require valid-user | Require valid-user | ||
+ | | ||
+ | # Or require local IP address (no authentication) | ||
+ | Require ip 192.168.0.0/ | ||
</ | </ | ||
</ | </ | ||
Line 1162: | Line 1166: | ||
$ egrep -Rl ' | $ egrep -Rl ' | ||
$ egrep -Rl ' | $ egrep -Rl ' | ||
- | $ egrep -Rl 'header(\' | + | # search for "header(' |
+ | $ egrep -Rl ' | ||
$ egrep -Rl ' | $ egrep -Rl ' | ||
+ | $ egrep -Rl ' | ||
$ find /var/www -newermt " | $ find /var/www -newermt " | ||
</ | </ | ||
Line 1187: | Line 1193: | ||
# Show files with pattern and contextual strings | # Show files with pattern and contextual strings | ||
$ cd /var/www | $ cd /var/www | ||
- | $ find . -name " | + | $ find . -name " |
- | $ find . -name " | + | $ find . -name " |
- | $ find . -name " | + | $ find . -name " |
- | $ find . -name " | + | $ find . -name " |
- | $ find . -name " | + | $ find . -name " |
- | $ find . -name " | + | $ find . -name " |
- | $ find . -name " | + | $ find . -name " |
- | $ find . -name " | + | $ find . -name " |
+ | $ find . -name " | ||
+ | $ find . -name " | ||
+ | |||
+ | # find ' | ||
+ | $ find . \( -name tcpdf -prune \) -o \( -name mpdf -prune \) -o \( -name fonts -prune \) -o -name " | ||
</ | </ | ||
Line 1263: | Line 1274: | ||
References: | References: | ||
* [[https:// | * [[https:// | ||
+ | == Troubleshooting == | ||
+ | ==== Random Freezes ==== | ||
+ | Under heavy IO load on servers you may see something like: | ||
+ | |||
+ | INFO: task xxxx blocked for more than 120 seconds. | ||
+ | | ||
+ | Eg. in ''/ | ||
+ | INFO: task nfsd:2252 blocked for more than 120 seconds. | ||
+ | "echo 0 > / | ||
+ | | ||
+ | ...typicall followed by a call trace that mentions the filesystem, and probably io_schedule and sync_buffer. | ||
+ | |||
+ | This message is not an error. | ||
+ | |||
+ | It is an indication that a program has had to wait for a very long time, and what it was doing. (which is not so informative of the reason - it's common that the real IO load issue comes from another process) | ||
+ | |||
+ | The code behind this sits in hung_task.c and was added somewhere around 2.6.30. This is a kernel thread that detects tasks that stays in the D state for a while (which typically meaning it is waiting for IO). | ||
+ | |||
+ | It complains when it sees a process has been waiting on IO so long that the whole process has not been scheduled for any CPU-time for 120 seconds (default). | ||
+ | |||
+ | Notes: | ||
+ | |||
+ | * If it happens constantly, your IO system is slower than your IO use. | ||
+ | * Most likely to happen to a process that was ioniced into the idle class. Which means it's working, idle-class is meant as an extreme politeness thing. It just indicates something else is doing a bunch of IO right now (for at least 120 seconds), e.g. '' | ||
+ | * If it happens only nightly, look at your cron jobs. | ||
+ | * A trashing system can cause this, and then it is purely a side effect of one program using too much RAM being blocked by a desktop-class drive with bad sectors (because they retry for a long while). | ||
+ | * NFS seems to be a common culprit, probably because it is good at filling the writeback cache, something which implies blocking while writeback happens - which is likely to block various things related to the same filesystem. (verify) | ||
+ | * If it happens on a fileserver, you may want to consider spreading to more fileservers, | ||
+ | * tweaking the linux io scheduler for the device may help (See Computer_data_storage_-_General_& | ||
+ | * If your load is fairly sequential, you may get some relief from using the noop io scheduler (instead of cfq) though note that that disables ionice). | ||
+ | * If your load is relatively random, upping the queue depth may help. | ||
+ | |||
+ | SOURCE: | ||
+ | * [[https:// | ||
+ | |||
+ | |||
+ | ==== Fix kernel panic error "task * blocked for more than 120 seconds" | ||
+ | |||
+ | If your server goes down, and you get a message like: | ||
+ | Feb 16 03:00:12 server kernel: INFO: task httpd:16101 blocked for more than 120 seconds. | ||
+ | |||
+ | Verify memory usage: | ||
+ | $ sar -r | ||
+ | |||
+ | Verify CPU usage: | ||
+ | $ sar -u | ||
+ | |||
+ | For me the culprit was CPU reaching %idle reaching 99.18 | ||
+ | < | ||
+ | 01:40:01 PM | ||
+ | ... | ||
+ | 06:30:01 PM | ||
+ | 06:40:01 PM | ||
+ | 06:50:01 PM | ||
+ | 07:00:01 PM | ||
+ | 07:10:01 PM | ||
+ | 07:20:01 PM | ||
+ | 07:30:01 PM | ||
+ | 07:40:01 PM | ||
+ | 07:50:01 PM | ||
+ | 08:00:01 PM | ||
+ | Average: | ||
+ | |||
+ | 08:06:39 PM LINUX RESTART | ||
+ | |||
+ | 08:10:01 PM | ||
+ | 08:20:01 PM | ||
+ | Average: | ||
+ | </ | ||
+ | |||
+ | Temporarily apply new settings for a couple of days. See here for a detailed explanation. | ||
+ | $ sudo sysctl -w vm.dirty_ratio=10 | ||
+ | $ sudo sysctl -w vm.dirty_background_ratio=5 | ||
+ | $ sudo sysctl -w vm.swappiness=60 | ||
+ | |||
+ | If everything runs smoothly, make the changes permanent: | ||
+ | $ vi / | ||
+ | |||
+ | Enter the following: | ||
+ | vm.dirty_ratio = 10 | ||
+ | vm.dirty_background_ratio = 5 | ||
+ | vm.swappiness = 60 | ||
+ | | ||
+ | Load settings from file / | ||
+ | $ sudo sysctl -p | ||
+ | |||
+ | |||
+ | SOURCE: | ||
+ | * [[https:// | ||
+ | * [[https:// | ||
+ | * [[https:// | ||
+ | |||
+ | |||
== References == | == References == |