6Jan/120

Removing old kernel to free up /boot partition

Posted by Rianto Wahyudi

For those who allocate less than 100MB for /boot on RHEL6 .. beware !
The kernel that ships with RHEL6 is quite big and you'll run out of space after few patches.

To list  all old kernel while excluding  currently active kernel:

 rpm -qa | egrep kernel-'(2|3|PAE|dev)' | sort -n | grep -v `uname -r| sed 's/PAE//'` 

Then run the following to remove it :

 yum remove $(rpm -qa | egrep kernel-'(2|3|PAE|dev)' | sort -n | grep -v `uname -r| sed 's/PAE//'` )
28Apr/110

MySource Matrix performance analysis : Log dump all database SQL queries

Posted by Rianto Wahyudi

MySource Matrix

Wanted to know why your mysource matrix installation running slow ?One major issue with our installation is  matrix execute large  number of SQL queries. The individual query by itself is very lightweight and executed fairly quick, but there are hundreds to thousands of queries required just to bring up a page.

I've experienced up to 5,000 SQL queries (  2 Mega Bytes in size !) just to bring up  2 KB html page. In this case,  powerful , grunty and well tuned  database server  just won't help improving  page load time. ( Network latency between app server and database server is  0.6 ms at best, so theoretical minimum page load time is 3 seconds. On top of that, there are databases and applications processing overhead. If combination of the database and application overhead takes around 0.5 millisecond to process, the page load time will be around 5.5 seconds. 5.5 seconds to bring up 1 html page with no assets  is too long !)

 

 

 

 

 

If you curious  to see these how many queries your matrix produce, you can use log_dump tools to log these queries into system.log To do this ,edit the  following file :

core/lib/MatrixDAL/MatrixDAL.inc

Add following  line at the top of the line :

require_once ('/fudge/dev/dev.inc');

Add log_dump() function at the end of  preparePdoQuery function ( Around line 285 - just before "return $query" )


log_dump($query);
return $query;

Output will be written to'/data/private/logs/system.log

UPDATE :

Squiz Matrix 4.4.1 ships with awesome "Performance Mode".  It can be accessed just by appending /_performance at the end of  the URL of your matrix site.

From : http://www.squizlabs.com/squiz-matrix/squiz-matrix-performance-mode

Performance Mode will analyse the performance of the currently viewed asset present its findings at the base of your browser

These performance results will report the total time taken to load the asset and how much of this time can be attributed to system processing; the remaining time due to individual asset loading.

The individual assets that have been loaded will be listed chronologically, displaying their name, asset ID and asset type, as shown in the image below.

mysource matrix squiz performance
10Apr/110

Purge squid cache using curl (without squidclient)

Posted by Rianto Wahyudi

Squid Logo Purging cached document from squid can easily be achieved using squidclient. But in some cases where squidclient is not available you can use curl to purge squid cache.

Purging squid cache is relatively simple, as you only need to specify PURGE http method instead of GET or PUT.

Your need to allow your web server   to PURGE data from squid. This is achievable  by specifying  ACL on squid server to allow web server , ie :

acl purgehosts src 192.168.1.10
http_access allow PURGE purgehosts

Bash

curl -X PURGE -x [squid-server:port] [URL to purge] -Is | head -n1

Example :

curl -X PURGE -x http://squid.example.com:80 http://www.example.com/contact -Is | head -n1

PHP

<!--?<span class="hiddenSpellError" pre=""-->php
$proxy="http://squid.example.com:80";
echo clearcache("http://www.example.com/contact",$proxy);

function clearcache($url,$proxy)
{
    if (!function_exists('curl_init')) { die('cURL is not installed!'); }

    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_PROXY, $proxy);
    curl_setopt($ch, CURLOPT_CUSTOMREQUEST , "PURGE");
    curl_setopt($ch, CURLOPT_TIMEOUT, 5);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
    curl_setopt($ch, CURLOPT_HEADER, true);
    curl_setopt($ch, CURLOPT_NOBODY, true);

    if (!curl_exec($ch)) { return false; }
    $output = curl_getinfo($ch, CURLINFO_HTTP_CODE);
    curl_close($ch);

    return ($output == '200') ? "success": "not cached";
}

?>
Filed under: Linux No Comments
29Mar/110

Postfix for DEV setup – Catch all email and forward it to a specific address

Posted by Rianto Wahyudi

Postfix Logo

On one hand we had a situation where DEV or QAS server accidentally sending email to various recipient. On the other hand we have developers and QA tester wanted to test and make sure that email functionality works. On top of all of these, the admin guys still want to receive their email coming from the server ie cron and logwatch email.

The following setup will configure postfix to accommodate all the requirements above. Postfix will prevent all outgoing email (black hole) . Instead of just dropping the email we capture and send a blind carbon copy (bcc) to a specific email address, leaving the “TO Address” intact. It will also allow selected mail to be routed to the original owner as well as the specific account receiving it.

Here is the config :

master.cf :

Add following at the end of master.cf . This will create a “discard” service.

discard   unix  -       n       n       -       -       pipe
user=nobody argv=/bin/cat /dev/null

main.cf :

Add the folllowing lines. All 3 lines are important. The first line is to configure BCC address. The second line is configure the email routing. The last line specifies that all domains are virtual domain for this postfix mail server.

recipient_bcc_maps = pcre:/etc/postfix/recipient_bcc
transport_maps = pcre:/etc/postfix/transport
virtual_alias_domains =

recipient_bcc:

# BCC all email to :
/^(.*)/       catchallemail@example.com

transport:

/catchallemail@example.com/    smtp:mailgateway.example.com
/administrator@example.com/	 smpt:mailgateway.example.com
/.*localdomain$/    local:
/.*/    discard:

Ideas developed from :
http://www.somerandomstuff.com/2009/07/09/howto-mail-blackhole/
http://www.wiredfool.com/2002/06/11/how-to-blackhole-email-server/ 

2Jan/110

Pingdom Update Contact for On Call Roster

Posted by Rianto Wahyudi


#!/bin/bash
# Script to update contact details on pingdom
# Accept phone number as the first argument
# Pingdom will automatically add +61[mobile number] - so the zero '0' prefix in front of mobile number need to be stripped
# Valid input : 432123456
# No error checking on this script as 'we know what we are doing(tm)'

# Change Log:
# 20100809 (rwahyudi) - Create script
# 20100820 (rwahyudi) - Add simple check

USERNAME=username@company.com
PASSWD=XXXXXX
MAIL=oncall-update@company.com
DIR=/opt/scripts/pingdom

if [ $# -lt 1 ]; then
echo "Usage: $0 [mobile number]"
echo " Pingdom will automatically add +61[mobile number] - so the zero '0' prefix in front of mobile number need to be stripped"
echo "Example: $0 432123456"
exit 1
fi

# Remove Cookie and login
rm $DIR/cookie.txt
curl -x proxy.company.com:8080 -c $DIR/cookie.txt -d "username=$USERNAME&amp;password=$PASSWD " https://pp.pingdom.com/index.php/login

# Edit OnCall contact
# OnCall ID = 12345
# See posting on how to get this value
curl -x proxy.company.com:8080 -b $DIR/cookie.txt -d "name=OnCall&amp;email=OnCall%40company.com&amp;country=15&amp;phone=$1&amp;twitter_dm=1&amp;twitter_user=&amp;defaultprovider=3&amp;retryprovider=1&amp;Submit=Update+Contact" https://pp.pingdom.com/index.php/member/contacts/edit/12345

# Check contact and email details on update to confirm that contact has been updated
PINGDOM_ACTIVE_NUMBER=$(curl -sq -b $DIR/cookie.txt https://pp.pingdom.com/index.php/member/contacts | grep -A1 " OnCall

"  | grep "+61-" | cut -d '-' -f2 | cut -d'&lt;' -f1)
PINGDOM_ACTIVE_ONCALL=$(grep $PINGDOM_ACTIVE_NUMBER $DIR/phonelist.csv  | cut -d ',' -f1)

echo "
`date`

Pingdom active On-Call : $PINGDOM_ACTIVE_ONCALL

"| mail -s "$PINGDOM_ACTIVE_ONCALL ($CRT_ACTIVE_ONCALL) is now the active On-Call contact"  $MAIL
27Nov/100

Bash script : DNS Check match between reverse and forward lookup

Posted by Rianto Wahyudi

Basic bash lookup script to check if forward / reverse IP address is matching.

  • The script use host command, it'll  automatically add DNS prefix based on /etc/resolv.conf
  • Support for hostnames  or PTR address with multiple RR
./checkhost
Usage: ./checkhost <IP or hostname> checkhost<IP or hostname> ...
Example: ./checkhost 69.163.128.215 www pop3.iinet.net.au
This script will check if forward and reverse match

./checkhost shadow pop3.iinet.net.au 202.138.0.9
shadow <-> 69.163.128.215 : Match
pop3.iinet.net.au <-> 203.0.178.192 : Match
mud.starway.net.au <-> 202.138.0.9 : Match

Download the script here :

checkhost
17Nov/101

NFS Monitoring Scripts

Posted by Rianto Wahyudi

I've tried few different NFS mount check, but most of them hang or freeze when the remote server is not responding. So I created a simple script to monitor and automatically mount NFS which kill it self .

I put this script on crontab which then run every 15 minutes :
cron entries :

*/15 * * * * /root/bin/check_nfs.sh

check_nfs.sh :

#!/bin/bash
# Script to check if NFS is mounted properly
# Change Log :

# Script to check and monitor NFS mounted file system

# When NFS is not working, it may hang the process that try to access it.
# As a work around, we start a process in the background that will kill nfs_check script after 30 seconds
# However, if check command works then kill the process that was about to kill me ( killmyparrent.sh )

REMOTE_SERVER=remotenfsserver.example.com
mnt_partition=/remote/nfs/folder/

# Exit if another instance is running
if [ `ps auxwww | grep check_nfs.sh | wc -l` -gt 3 ]
then
echo `pgrep check_nfs.sh | wc -l`
exit
fi

if mount | grep -q $mnt_partition
then :

/root/bin/killmyparent.sh $ "Email Subject - eg : NFS mount failed on xxx server" "Some issue with $mnt_parition. Please check it will you !" &
killerpid=`pgrep killmyparent.sh`

# NFS file system appear to be mounted - lets check if we can access it ..
if df | grep -q $mnt_partition
then :
# df command works .. kill "killmyparent.sh" script before it kill us
kill $killerpid
exit 1;
fi
else
# Wait until server is contactible and then mount the partition
until [ `ping -q -c3 -w5 $REMOTE_SERVER > /dev/null 2>&1; echo $?` -eq 0 ]
do
sleep 10;
done
mount $mnt_partition
exit $?
fi

killmyparent.sh

#!/bin/bash
if [ -z $1 ]
then
echo "Usage $0 "
exit
fi
sleep 20
kill -9 $1
echo $3 | mail -s $2
15Nov/101

Bind 9.2 RNDC Stats

Posted by Rianto Wahyudi

Installed bind DNS statistics last week by following  : http://jpastuszek.sigquit.net/2009/03/graphing-bind-query-statistics-with.html

The graphs looks pretty !

Bind DNS Statistics

DNS Statistics

success The number of successful queries made to the server or zone. A successful query is defined as query which returns a NOERROR response with at least one answer RR.
referral The number of queries which resulted in referral responses.
nxrrset The number of queries which resulted in NOERROR responses with no data.
nxdomain The number of queries which resulted in NXDOMAIN responses.
failure The number of queries which resulted in a failure response other than those above.
recursion The number of queries which caused the server to perform recursion in order to find the final answer.

Each query received by the server will cause exactly one of successreferralnxrrsetnxdomain, or failureto be incremented, and may additionally cause the recursion counter to be incremented.

Reference : http://www.networking-tips.net/?p=85

Nxdomain and NXrrset are new terms for me.

NXDOMAIN :

NXDOMAIN or Non-Existent Domain is a term used for the Internet domain name that is unable to be resolved using the DNS implementation owing either to domain name not yet registered or the server problem.

The reference to the NXDOMAIN is published in RFC 1035 (Domain names - implementation and specification) and also in RFC 2308 (Negative Caching of DNS Queries or referred to as DNS NCACHE).

Reference : http://it.toolbox.com/wiki/index.php/NXDOMAIN

NXRRSET :

Simple explanation  - NXRRSET means no such record set. The domain exist but record set does not exist. Look at the named cache dump below, it's clear that AAAA record for www.asx.com.au does not exist :

asx.com.au.             7293    NS      ns3.asx.com.au.
7293    NS      ns01.asx.com.au.
7293    NS      ns02.asx.com.au.
; additional
ns01.asx.com.au.        7293    A       203.15.147.70
; additional
ns02.asx.com.au.        7293    A       203.15.147.74
; glue
ns1.asx.com.au.         10893   A       203.15.147.70
; glue
ns2.asx.com.au.         10893   A       203.15.147.70
; glue
ns3.asx.com.au.         10893   A       203.15.147.74
; authauthority
www.asx.com.au.         93      \-AAAA  ;-$NXRRSET
; authanswer
7293    A       203.15.147.66
; authauthority
www9.asx.com.au.        99      \-AAAA  ;-$NXRRSET

Detailed information can be found : http://www.ietf.org/rfc/rfc2136.txt

11Nov/100

Hello world!

Posted by Rianto Wahyudi

Welcome to WordPress. This is your first post. Edit or delete it, then start blogging!

Filed under: Uncategorized No Comments