Lots of files to remove?

Date01 Dec 2010 ContentComments

So I recently have a few hundred thousand files in /tmp/ (was testing something and needed to remove them all)!

Bash as always is very helpful

[root@localhost]# rm -rf /tmp/*
-bash: /bin/rm: Argument list too long”

You also have this issue? Don’t worry you can use find and feed the results into rm as follows

[root@localhost]# find /tmp/ | xargs rm

(You could also specify -type f or -type d if you only wanted files or directory, you may need rm -r rather than rm if you want to recurse into directories)

If you have seen the output of find previously you will know why this works, if not then here is a quick snippet:

[root@localhost]# find /tmp/
/tmp/repoup1291034701
/tmp/repoup1291034701/live
/tmp/repoup1291034701/live/.git

xargs just converts from standard input to command line arguments as can be seen below:

[root@localhost]# find /tmp/ | head -n 1 | xargs echo rm
rm /tmp/repoup1291034701

Hope this helps

Compiling nginx and PHP-FPM on CentOS 5

Date29 Nov 2010 ContentComments

Luke Carrier, founder and lead developer at CloudFlux recently wrote a how-to on compiling Nginx and PHP-FPM (can be found here). Nginx is well known for being a high-performance HTTP server and reverse proxy, in fact this very website is running on-top of Nginx. A few websites you might know that run Nginx are:

These sites show how Nginx can be deployed in many high traffic configurations, GitHub being well know for the use of RoR, Wordpress the use of PHP etc. Whether you are deploying PHP under FPM, RoR under Passenger, static files or data from memcached Nginx can outperform many other web servers.

The main advantage to using Nginx is it’s event-driven (asynchronous) architecture, this means that alongside providing high-performance there is a small predictable memory footprint. All the features make the server a good performer in all environments from single VPS systems to web clusters.

Some features you’ll probably love:

  • Modules for load balancing

  • Direct memcached integration

  • Simple proxy configuration

  • Passenger integration

  • Ability to serve over 50,000 simultaneous connections

  • Statistics module

  • Low memory footprint

  • Clean simple to edit configuration files

  • Zero downtime configuration and binary updates

  • PCRE based rewrite rules

As you can see from their site Nginx as a server is highly regarded and recommended. I’d suggest taking a look, having a play and revealing what it can do for you.

Powershell!?!?!?!

Date21 Jul 2010 ContentComments

Most of been bored last night, decided to have my first bash at automation in windows using powershell, kinda impressed but now where near the level that you can get in *nix :D Hack of an attempt can be seen below (it works mainly but is pretty untested for the function actions as I don’t have a windows server that can run hyperv, looks right per the docs though!):

# Settings

## Email

$email_system = ""

$email_contact = ""

$email_host = ''

$email_username = ''

$email_password = ''
## Database

$database_host = ""

$database_name = ""

$database_user = ""

$database_pass = ""
## Run task settings

$run_tasks = 1

$run_reboots = 1

$run_poweroffs = 1

$run_powerups = 1
# Log file directory

$file_log_dir = 'C:\automation_logs\'
#

#

#

# Don't touch below here

#

#

#
# Session varibles
$base_log_name = $file_log_dir+$(Get-Date -format 'd-M-y-h-m-s')+'-'
# Functions
## Core functions

function log([string]$message, [string]$type = 'info')

{

if ((Test-Path -path $file_log_dir) -ne $True){

# Make log dir if it dosn't exist

write-host 'Making log dir'

New-Item $file_log_dir -type directory

}
$log_file = $base_log_name+$($type)+'.log'

write-host $message

add-content $log_file $message

}
function error([string]$traceback)

{

$error = 'Hit exception:'+$traceback

log $error 'error'

mail 'Exception encountered' $error

}
function mail([string]$subject, [string]$message)

{

#$smtp_handler = new-object Net.Mail.SmtpClient($email_host)

#$smtp_handler.Credentials = New-Object System.Net.NetworkCredential($email_username, $email_password)

#$smtp_handler.Send($email_system, $email_contact, $subject, $message)

}
function marktaskFailed([int]$task_id)

{

$time = [int][double]::Parse((Get-Date -UFormat %s))
try {
 $command = $database_handler.CreateCommand()
 $command.CommandText = "update queue set status = 'f', run_time = '$($time)' where id = '$($task_id)'"

$command.ExecuteNonQuery()

} catch {

log "Could not set status = failed for task id $task_id"

error $_.Exception.ToString()

}

}
function marktaskSuccess([int]$task_id)

{

$time = [int][double]::Parse((Get-Date -UFormat %s))
try {

$command = $database_handler.CreateCommand()

$command.CommandText = "update queue set status = 'c', run_time = '$($time)' where id = '$($task_id)'"

$command.ExecuteNonQuery()

} catch {

log "Could not set status = complete for task id $task_id"

error $_.Exception.ToString()

}

}
## Action functions

function rebootHyperVInstance($task)

{

# Un-tested bit but this is right according to the bizzare docs

$vm = gwmi -namespace root\virtualization -query "SELECT * FROM Msvm_ShutdownComponent WHERE SystemName = '$($task['machine_id'])'"

$result = $vm.InitiateShutdown("$true", "Automated shutdown requested")
if($result.returnvalue -match "0"){

$vm = gwmi -namespace root\virtualization -query "SELECT * FROM msvm_computersystem WHERE SystemName = '$($task['machine_id'])'"

$result = $vm.requeststatechange(2)
if($result.returnvalue -match "0"){

log "Task '$($task['task_id'])' ($($task['type'])) completed with success status"

marktaskSuccess($task['task_id'])

}else{

log "Task '$($task['task_id'])' ($($task['type'])) completed with failed status, could not issue powerup"

marktaskFailed($task['task_id'])

}

}else{

log "Task '$($task['task_id'])' ($($task['type'])) completed with failed status, could not shutdown vm"

marktaskFailed($task['task_id'])

}

}
function poweroffHyperVInstance($task)

{

# Un-tested bit but this is right according to the bizzare docs

$vm = gwmi -namespace root\virtualization -query "SELECT * FROM Msvm_ShutdownComponent WHERE SystemName = '$($task['machine_id'])'"

$result = $vm.InitiateShutdown("$true", "Automated shutdown requested")
if($result.returnvalue -match "0"){

log "Task '$($task['task_id'])' ($($task['type'])) completed with success status"

marktaskSuccess($task['task_id'])

}else{

log "Task '$($task['task_id'])' ($($task['type'])) completed with failed status, could not issue reboot"

marktaskFailed($task['task_id'])

}

}
function poweronHyperVInstance($task)

{

# Un-tested bit but this is right according to the bizzare docs

$vm = gwmi -namespace root\virtualization -query "SELECT * FROM msvm_computersystem WHERE SystemName = '$($task['machine_id'])'"

$result = $vm.requeststatechange(2)
if($result.returnvalue -match "0"){

log "Task '$($task['task_id'])' ($($task['type'])) completed with success status"

marktaskSuccess($task['task_id'])

}else{

log "Task '$($task['task_id'])' ($($task['type'])) completed with failed status, could not issue powerup"

marktaskFailed($task['task_id'])

}

}

 

# Main code
## Load what we need!

try{

[void][System.Reflection.Assembly]::LoadWithPartialName("MySql.Data")

} catch {

log "Could not load mysql.net connector"

error $_.Exception.ToString()

break

}
## Main loop

while (1) {

write-host 'Doing run....'
## Connect to the database

try{

$database_handler = New-Object MySql.Data.MySqlClient.MySqlConnection

$database_handler.ConnectionString = "server=$database_host;uid=$database_user;pwd=$database_pass;database=$database_name;"

$database_handler.Open()

} catch {

log "Could not connect to the database :("

error $_.Exception.ToString()

break

}
write-host 'Connected to database'
try {

$command = $database_handler.CreateCommand()

$command.CommandText = "select * from `queue` where `status` = 'n'"

$data = $command.ExecuteReader()

} catch {

log "Could not run query against database"

error $_.Exception.ToString()

continue

}
write-host 'Got queue from database'
# Array where we will put our tasks!

$tasks_to_run = @()
write-host 'Loading queue'
while ($data.Read()) {

# task

try {

$task = @{}

$task['type'] = $data.GetValue(1).ToString()

$task['task_id'] = $data.GetValue(0).ToString()

$task['machine_id'] = $data.GetValue(3).ToString()
# Add task to list of tasks to run

$tasks_to_run += $task

write-host "Entered task $($task['task_id']) into queue"

} catch {

log "Could not add task to queue"

error $_.Exception.ToString()

continue

}

}

$data.close() # Close the data reader

write-host 'Loaded queue'
if($tasks_to_run.length -gt 0){

write-host "Tasks to run: $($tasks_to_run.length)!"
# Loop though the list of tasks to run

for ( $i = 0 ; $i -lt $tasks_to_run.length; $i++ ){

$task = $tasks_to_run[$i]
log "Running task $($task['task_id']) (Action '$($task['type'])' for instance '$($task['machine_id'])')"
 if($run_tasks -eq 1){

# The bit that decides what we are going to do

switch ($task['type']) {

poweron {

if($run_powerups -eq 1){

&poweronHyperVInstance($task)
 }else{
 log "Not running task '$($task['task_id'])', powerup tasks are disabled"
 marktaskFailed($task['task_id'])
 }

}
poweroff {
 if($run_poweroffs -eq 1){

&poweroffHyperVInstance($task)
 }else{
 log "Not running task '$($task['task_id'])', poweroff tasks are disabled"
 marktaskFailed($task['task_id'])
 }

}
reboot {
 if($run_reboots -eq 1){

&rebootHyperVInstance($task)
 }else{
 log "Not running task '$($task['task_id'])', reboot tasks are disabled"
 marktaskFailed($task['task_id'])
 }

}
default { # We were asked to do something we don't understand!

log "Tried to run an unknown action: '$($task['type'])' for task '$($task['task_id'])' this will need running manually!"

mail "Tried to run an unknown action: '$($task['type'])' for task '$($task['task_id'])' this will need running manually!"

marktaskFailed($task['task_id'])

}

}
 }else{
 log "Not running task '$($task['task_id'])', tasks are disabled"

marktaskFailed($task['task_id'])
 }

}

write-host 'Tasks run'

}else{

write-host "No tasks to run!"

}
write-host 'Run complete....'

# Sleep 10 seconds until next run

Start-Sleep 10

}

Anyway back to making this CSS I’m working on look half decent in IE!

virt-install is a life server

So on one of my many servers that sit around doing nothing all day I have KVM installed, recently I’ve been playing about with this quite a bit. Still trying to figure out libvirt and the api to it which is like a black hole when it comes to documented functions as the Python classes (which is what I’m using) are generated pretty much straight off the C api stuff. Anyway as I was playing with that and rrdtool trying to make a cool graphing utility, I got bored with having to type loads to install a new dom. So thanks to the power of lighttpd to serve my kickstats, virt-install to do the leg work and the amazing thing called lvm I hacked up a bash script to do the hard work for me. Find it as follows:

#!/bin/bash
VG='/dev/virtual_disks'
KS_ADDR='http://192.168.100.1:5843/kickstarts'

case "$1" in
 centos)
 test -z $2 && exit 5
 test -z $3 && exit 5
 test -z $4 && exit 5
 if [ ! -z $5 ]; then
 extra="auto=true hostname=$4 domain='' ks=$KS_ADDR/centos_$5.ks"
 else
 extra="auto=true hostname=$4 domain=''"
 fi
 echo "Installing vm $4 with $2G disk and $3 ram allocation passing kernel arguments '$extra'"
 location='http://mirror.limestonenetworks.com/centos/5.5/os/x86_64/'
 lvcreate --name $4 --size $2G virtual_disks
 virt-install --connect qemu:///system --name $4 --ram $3 --hvm --file $VG/$4 --vnc --location=$location --noautoconsole --extra="$extra"
 virsh vncdisplay $4
 ;;
 fedora)
 test -z $2 && exit 5
 test -z $3 && exit 5
 test -z $4 && exit 5
 if [ ! -z $5 ]; then
 extra="auto=true hostname=$4 domain='' ks=$KS_ADDR/fedora_$5.ks"
 else
 extra="auto=true hostname=$4 domain=''"
 fi
 echo "Installing vm $4 with $2G disk and $3 ram allocation passing kernel arguments '$extra'"
 location='http://www.mirrorservice.org/sites/download.fedora.redhat.com/pub/fedora/linux/releases/13/Fedora/x86_64/os/'
 lvcreate --name $4 --size $2G virtual_disks
 virt-install --connect qemu:///system --name $4 --ram $3 --hvm --file $VG/$4 --vnc --location=$location --noautoconsole --extra="$extra"
 virsh vncdisplay $4
 ;;
 debian)
 test -z $2 && exit 5
 test -z $3 && exit 5
 test -z $4 && exit 5
 if [ ! -z $5 ]; then
 extra="auto=true hostname=$4 domain='' url=$KS_ADDR/debian_$5.ks"
 else
 extra="auto=true hostname=$4 domain=''"
 fi
 echo "Installing vm $4 with $2G disk and $3 ram allocation passing kernel arguments '$extra'"
 location='http://debian.intergenia.de/debian/dists/lenny/main/installer-amd64/'
 lvcreate --name $4 --size $2G virtual_disks
 virt-install --connect qemu:///system --name $4 --ram $3 --hvm --file $VG/$4 --vnc --location=$location --noautoconsole --extra="$extra"
 virsh vncdisplay $4
 ;;
 ubuntu)
 test -z $2 && exit 5
 test -z $3 && exit 5
 test -z $4 && exit 5
 if [ ! -z $5 ]; then
 extra="auto=true hostname=$4 domain='' url=$KS_ADDR/ubuntu_$5.ks"
 else
 extra="auto=true hostname=$4 domain=''"
 fi
 echo "Installing vm $4 with $2G disk and $3 ram allocation passing kernel arguments '$extra'"
 location="http://ubuntu.intergenia.de/ubuntu/dists/lucid/main/installer-amd64/"
 lvcreate --name $4 --size $2G virtual_disks
 virt-install --connect qemu:///system --name $4 --ram $3 --hvm --file $VG/$4 --vnc --location=$location --noautoconsole --extra="$extra"
 virsh vncdisplay $4
 ;;
 *)
 echo $"Usage: $0 {centos|fedora|debian|ubuntu} {Disk Size} {Ram Allocation} {Name} {Kickstart}"
 exit 2
 ;;
esac

Example usage:

[root@virtual1 ~]# ./vm_install.sh fedora 50 1024 CharityHosting1 basic
Installing vm CharityHosting1 with 50G disk and 1024 ram allocation passing kernel arguments 'auto=true hostname=CharityHosting1 domain='' ks=http://192.168.100.1:5843/kickstarts/fedora_basic.ks'
 Logical volume "CharityHosting1" created
Starting install...
Retrieving file .treeinfo... | 2.4 kB 00:00 ...
Retrieving file vmlinuz... | 6.7 MB 00:01 ...
Retrieving file initrd.img... | 57 MB 00:15 ...
Creating domain... | 0 B 00:00
Domain installation still in progress. You can reconnect to
the console to complete the installation process.
:0

The :0 thrown out at the end is the VNC port you can connect to which is pretty much the console on the vm. Bear in mind this does very very little validation, if a domain is already defined it will error out at that stage so your pretty safe but for example if you tell it to use a lvm that already exists it will error but carry on installing and wipe everything off :-)

2 Variables are as follows: VG - This is the volume group the lvm’s are going to reside in, the path is the format that you would use when calling lvcreate (duh) KS_ADDR - This is the base http path to where the kick-starts are located, I personally use lighttpd to do this basic job. All kickstats have the format $distro”_“$name”.ks”

The main work is done in the kick-start file, at the moment they are pretty static but eventually will be built dynamically out of a database so you can customize things like the root password, set static ips, install packages etc without having to write out a custom one.

The other thing I do along side this is make dnsmasq hand out “static” ips to servers because I use NAT for mapping the public to private addresses, this is because I only run a couple of production vms on that server (Such as the charity hosting servers and graphing server) and the rest are test servers with no need for public addresses.

Setup for dnsmasq/the script to make the assignments static are as follows: DNSMASQ config:

domain-needed
bogus-priv
strict-order
bind-interfaces
listen-address=192.168.100.1
except-interface=lo
dhcp-range=192.168.100.2,192.168.100.254
dhcp-lease-max=253
conf-dir=/etc/dnsmasq.d/

/root/fix_leases.py script (this runs every 2min)

#!/usr/bin/python
import os
leases = {}
known_hosts = {}

if os.path.isfile('/var/lib/dnsmasq/dnsmasq.leases'):
 leases_fh = open('/var/lib/dnsmasq/dnsmasq.leases', 'r')
 for line in leases_fh.readlines():
 (time, mac, ip, hostname, clientid) = line.split(' ')
 leases[mac] = {
 'time': time,
 'ip': ip,
 'hostname': hostname,
 'client_id': clientid,
 }

leases_fh.close()

if os.path.isfile('/etc/dnsmasq.d/static_leases'):
 known_host_fh = open('/etc/dnsmasq.d/static_leases', 'r')
 for line in known_host_fh.readlines():
 if '#' in line:
 hostname = line.split('# Host')[1].strip()
 elif '=' in line:
 if hostname == '':
 hostname = 'Unknown'
 data = line.split('=')[1]
 (mac, ip) = data.split(',')
 known_hosts[mac] = {
 'ip': ip,
 'hostname': hostname,
 }
 hostname = ''

new_leases = {}
for mac in leases:
 if mac not in known_hosts:
 new_leases[mac] = leases[mac]

hosts = open('/etc/dnsmasq.d/static_leases', 'w')
for mac in known_hosts:
 ip = known_hosts[mac]['ip']
 hostname = known_hosts[mac]['hostname']
 hosts.write("# Host %s\ndhcp-host=%s,%s\n" % (hostname, mac, ip))

for mac in new_leases:
 ip = new_leases[mac]['ip']
 hostname = new_leases[mac]['hostname']
 hosts.write("# Host %s\ndhcp-host=%s,%s\n" % (hostname, mac, ip))

hosts.close()
print "Updating done :)"
os.system('service dnsmasq reload')

To make the above work I run dnsmasq outside of libvirt which is not the default (you have to enable dhcp in libvirt to do it) but it works pretty well for me, just have to clear the leases out every now and then ;-)

Until next time which may include some C as I’m currently working on a DNS management app for the iPhone (I will probably give up on it soon though due to time constrictions!)

Coding Help

Date09 Jun 2010 ContentComments

So I’m currently working on a small project which will require authentication. Not really sure how to achieve what I want so here’s a quick explanation for anyone if you want to share ideas: Basic SQL definition:

mysql>; describe users;
+---------------------------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------------------+------------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| username | varchar(500) | NO | | NULL | |
| password | varchar(100) | NO | | NULL | |
| salt | varchar(15) | NO | | NULL | |
| group_id | int(20) | NO | | NULL | |
| perm_login | set('Y','N','G') | NO | | G | |
+---------------------------+------------------+------+-----+---------+----------------+

mysql>; describe user_groups;
+---------------------------+---------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------------------+---------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(500) | NO | | NULL | |
| perm_login | set('Y','N') | NO | | Y | |
+---------------------------+---------------+------+-----+---------+----------------+

For permissions I want to do groups but then have the option to override them on a per user basis. All permissions fields are named perm_ , in the users table there are the Y/N as options then G, if the field is G then it uses the identical permissions field from the group table that the user is part of (group_id matches up with id in the user_groups table).

Basic SQL idea:

SELECT IF((`users`.`perm_test` = "G"), `user_groups`.`perm_test`, `users`.`perm_test`) AS allowed from users LEFT JOIN (`user_groups`) ON (`user_groups`.`id`=`users`.`group_id`) WHERE `users`.`username` = "testuser"

Now this works just fine however I can see issues mainly limitations to only being able to check one permission at a time. For 99% of the time this wouldn’t be an issue for example when using the web interface/api there would be at a minimum of 2 calls and most of the time that would be it however in certain cases where you needed a full list there would be large issues. For a start getting a list of permissions would be highly recourse intensive and returning everyone would result in running something like the above for every permission. This may be a few hundred times per user per load.

I’m thinking of a way to move permissions into there own table then call them in a relational fashion but cannot think how to do this keeping the user/group system. Any ideas would be massively appreciated, for the moment db scheme changes are fine as this is very much a testing structure.

Speak to you soon :)