Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
systems:linux_setup_and_configuration [2015/12/03 17:54]
smayr [Cleaning Hacked Server] 		systems:linux_setup_and_configuration [2019/07/29 14:27] (current)
ajdavis [Linux Setup <del>Outdated as of 2019</del>]
Line 1: Line 1:
-== Linux Setup ==+== Linux Setup (Outdated as of 2019) ==
 === Installation === === Installation ===
 # Download CD images (ISO files) from http://fedora.redhat.com # Download CD images (ISO files) from http://fedora.redhat.com
Line 55: Line 55:
 AuthUserFile /data/www/html/<protectedfolder>/.htpasswd AuthUserFile /data/www/html/<protectedfolder>/.htpasswd
 <Limit GET> <Limit GET>
 +        # Require an authenticated user
         Require valid-user         Require valid-user
 +        
 +        # Or require local IP address (no authentication)
 +        Require ip 192.168.0.0/255.255.255.0
 </Limit> </Limit>
 </code> </code>
Line 1162: Line 1166:
 $ egrep -Rl 'se_referrer' /var/www $ egrep -Rl 'se_referrer' /var/www
 $ egrep -Rl 'base64_decode' /var/www $ egrep -Rl 'base64_decode' /var/www
 +# search for "header('HTTP/1.0 404 Not Found\')"
 +$ egrep -Rl 'HTTP/1.0 404 Not Found' /var/www
 +$ egrep -Rl 'preg_replace("/.*/e",' /var/www
 +$ egrep -Rl 'long2ip",' /var/www
 $ find /var/www -newermt "yyyy-mm-dd" $ find /var/www -newermt "yyyy-mm-dd"
 </code> </code>
Line 1167: Line 1175:
 Search all PHP files for a condition: Search all PHP files for a condition:
 <code bash> <code bash>
-$ find /home/*/public_html/ -type f -name "*php" -exec egrep -l '\$GLOBALS.*\\x|function.*for.*strlen.*isset' "{}" \; +# Examples: 
-$ find /var/www -type f -name "*php" -exec egrep -l 'se-referrer' "{}" \; +# $ find /home/*/public_html/ ... 
-$ find /var/www -type f -name "*php" -exec egrep -l 'base64_decode' "{}" \; +# $ find /var/www ... 
-$ find /var/www -type f -name "*php" -exec egrep -l '<iframe' "{}" \;+# $ find ... -exec egrep -l 'pattern1|pattern2|pattern3.*pattern4.*pattern5' "{}" \; 
 + 
 +# Show file list of files with patterns 
 +$ cd /var/www 
 +$ find . -type f -name "*php" -exec egrep -Rl '\$GLOBALS.*\\x|function.*for.*strlen.*isset' "{}" \; 
 +$ find . -type f -name "*php" -exec egrep -Rl 'isset.*eval' "{}" \; 
 +$ find . -type f -name "*php" -exec egrep -Rl '<iframe' "{}" \; 
 +$ find . -type f -name "*php" -exec egrep -Rl 'PHPMailer' "{}" \; 
 +$ find . -type f -name "*php" -exec egrep -Rl 'encodeURIComponent' "{}" \; 
 +$ find . -type f -name "*php" -exec egrep -Rl 'se-referrer' "{}" \; 
 +$ find . -type f -name "*php" -exec egrep -Rl 'base64_decode' "{}" \; 
 +$ find . -type f -name "*php" -newermt "yyyy-mm-dd" 
 + 
 +# Show files with pattern and contextual strings 
 +$ cd /var/www 
 +$ find . -name "*.php" -exec grep --color -H 'function.*for.*strlen.*isset' "{}" \; 
 +$ find . -name "*.php" -exec grep --color -H '\$GLOBALS.*\\x' "{}" \; 
 +$ find . -name "*.php" -exec grep --color -H 'isset.*eval' "{}" \; 
 +$ find . -name "*.php" -exec grep --color -H 'PHPMailer' "{}" \; 
 +$ find . -name "*.php" -exec grep --color -H 'encodeURIComponent' "{}" \; 
 +$ find . -name "*.php" -exec grep --color -H 'se_referrer' "{}" \; 
 +$ find . -name "*.php" -exec grep --color -H 'base64_decode' "{}" \; 
 +$ find . -name "*.php" -exec grep --color -H 'header(*Location:' "{}" \; 
 +$ find . -name "*.php" -exec grep --color -H 'long2ip' "{}" \; 
 +$ find . -name "*.php" -exec grep --color -H 'ip2long' "{}" \; 
 + 
 +# find '404' in PHP files (excluding in dirs tcpdf, mpdf, and fonts) 
 +$ find . \( -name tcpdf -prune \) -o \( -name mpdf -prune \) -o \( -name fonts -prune \) -o -name "*.php" -exec grep --color -H '404'  "{}" \;
 </code> </code>
  
Line 1239: Line 1274:
 References:  References: 
   * [[https://wordpress.org/support/topic/link-templatephpsuspected/page/2]]   * [[https://wordpress.org/support/topic/link-templatephpsuspected/page/2]]
 +== Troubleshooting ==
 +==== Random Freezes ====
 + Under heavy IO load on servers you may see something like:
 +
 +  INFO: task xxxx blocked for more than 120 seconds.
 +  
 +Eg. in ''/var/log/syslog'':
 +  INFO: task nfsd:2252 blocked for more than 120 seconds.
 +  "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
 +  
 +...typicall followed by a call trace that mentions the filesystem, and probably io_schedule and sync_buffer.
 +
 +This message is not an error.
 +
 +It is an indication that a program has had to wait for a very long time, and what it was doing. (which is not so informative of the reason - it's common that the real IO load issue comes from another process)
 +
 +The code behind this sits in hung_task.c and was added somewhere around 2.6.30. This is a kernel thread that detects tasks that stays in the D state for a while (which typically meaning it is waiting for IO).
 +
 +It complains when it sees a process has been waiting on IO so long that the whole process has not been scheduled for any CPU-time for 120 seconds (default).
 +
 +Notes:
 +
 +  * If it happens constantly, your IO system is slower than your IO use.
 +  * Most likely to happen to a process that was ioniced into the idle class. Which means it's working, idle-class is meant as an extreme politeness thing. It just indicates something else is doing a bunch of IO right now (for at least 120 seconds), e.g. ''updatedb'' (may be victim if it were ioniced, cause if not).
 +  * If it happens only nightly, look at your cron jobs.
 +  * A trashing system can cause this, and then it is purely a side effect of one program using too much RAM being blocked by a desktop-class drive with bad sectors (because they retry for a long while).
 +  * NFS seems to be a common culprit, probably because it is good at filling the writeback cache, something which implies blocking while writeback happens - which is likely to block various things related to the same filesystem. (verify)
 +  * If it happens on a fileserver, you may want to consider spreading to more fileservers, or using a parallel filesystem.
 +  * tweaking the linux io scheduler for the device may help (See Computer_data_storage_-_General_&_RAID_performance_tweaking#OS_scheduling).
 +  * If your load is fairly sequential, you may get some relief from using the noop io scheduler (instead of cfq) though note that that disables ionice).
 +  * If your load is relatively random, upping the queue depth may help.
 +
 +SOURCE: 
 +  * [[https://helpful.knobs-dials.com/index.php/INFO:_task_blocked_for_more_than_120_seconds]]
 +
 +
 +==== Fix kernel panic error "task * blocked for more than 120 seconds" ====
 +
 +If your server goes down, and you get a message like:
 +  Feb 16 03:00:12 server kernel: INFO: task httpd:16101 blocked for more than 120 seconds.
 +
 +Verify memory usage:
 +  $ sar -r
 +
 +Verify CPU usage:
 +  $ sar -u
 +
 +For me the culprit was CPU reaching %idle reaching 99.18
 +<code>
 +01:40:01 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
 +...
 +06:30:01 PM     all     42.52      0.00      3.52      0.32      0.00     53.65
 +06:40:01 PM     all     43.72      0.00      3.60      0.32      0.00     52.36
 +06:50:01 PM     all     42.32      0.00      3.47      0.38      0.00     53.82
 +07:00:01 PM     all     41.38      0.00      3.47      0.25      0.00     54.91
 +07:10:01 PM     all     44.30      0.00      3.50      0.65      0.00     51.54
 +07:20:01 PM     all     36.22      0.00      2.89      0.38      0.00     60.50
 +07:30:01 PM     all     31.56      0.00      2.72      0.26      0.00     65.47
 +07:40:01 PM     all     23.77      0.00      2.03      0.41      0.00     73.79
 +07:50:01 PM     all      2.01      0.00      0.20      0.14      0.00     97.65
 +08:00:01 PM     all      0.59      0.00      0.07      0.15      0.00     99.18
 +Average:        all     36.88      0.10      3.22      0.39      0.00     59.41
 + 
 +08:06:39 PM       LINUX RESTART
 + 
 +08:10:01 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
 +08:20:01 PM     all     50.79      0.00      3.62      0.08      0.00     45.51
 +Average:        all     50.79      0.00      3.62      0.08      0.00     45.51
 +</code>
 +
 +Temporarily apply new settings for a couple of days. See here for a detailed explanation.
 +  $ sudo sysctl -w vm.dirty_ratio=10
 +  $ sudo sysctl -w vm.dirty_background_ratio=5
 +  $ sudo sysctl -w vm.swappiness=60
 +
 +If everything runs smoothly, make the changes permanent:
 +  $ vi /etc/sysctl.conf
 +
 +Enter the following:
 +  vm.dirty_ratio = 10
 +  vm.dirty_background_ratio = 5
 +  vm.swappiness = 60
 +  
 +Load settings from file /etc/sysctl.conf:
 +  $ sudo sysctl -p
 +
 +
 +SOURCE: 
 +  * [[https://duntuk.com/fix-kernel-panic-error-task-blocked-more-120-seconds]]
 +  * [[https://www.blackmoreops.com/2014/09/22/linux-kernel-panic-issue-fix-hung_task_timeout_secs-blocked-120-seconds-problem/]]
 +  * [[https://www.cyberciti.biz/faq/linux-kernel-tuning-virtual-memory-subsystem/]]
 +
 +
 == References == == References ==
systems/linux_setup_and_configuration.1449183263.txt.gz · Last modified: 2015/12/03 17:54 by smayr
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International


  • Show page
  • Old revisions
  • Backlinks
  • Recent Changes
  • Media Manager
  • Sitemap