In continuing working with XMPP and Python, I managed to get xmpppy working.

This proved to be particularly tricky because xmpppy uses some outdated modules like md5 and sha.  It also relies on some dns functions which no longer seem to work.

I was able to use the sample script "xsend.py" that the project provides.  The big thing is that I had to specify talk.google.com and the port number directly in the connect command.

If I did not specify the servername, I would get the error:
 An error occurred while looking up _xmpp-client._tcp.example.com

NOTE: DNS is configured correctly for the domain I was practicing with, but the xmpppy code was not able to resolve it.

To run the script, simply execute;

./xblast.py destination@example.com  text to send

#!/usr/bin/python
# $Id: xsend.py,v 1.8 2006/10/06 12:30:42 normanr Exp $
import sys,os,xmpp,time

if len(sys.argv) < 2:
    print "Syntax: xsend JID text"
    sys.exit(0)

tojid=sys.argv[1]
text=' '.join(sys.argv[2:])

jidparams={}

jidparams['jid']='system@example.com'
jidparams['password'] = '123456'

jid=xmpp.protocol.JID(jidparams['jid'])
cl=xmpp.Client(jid.getDomain(),debug=True)

con=cl.connect(('talk.google.com',5222),  use_srv=False)

if not con:
    print 'could not connect!'
    sys.exit()
print 'connected with',con
auth=cl.auth(jid.getNode(),jidparams['password'],resource=jid.getResource())
if not auth:
    print 'could not authenticate!'
    sys.exit()
print 'authenticated using',auth

# cl.SendInitPresence(requestRoster=0)   # you may need to uncomment this for old server
id=cl.send(xmpp.protocol.Message(tojid,text))
print 'sent message with id',id

time.sleep(1)   # some older servers will not send the message if you disconnect immediately after sending

cl.disconnect()


---

I found that someone has written a perl script geared towards sending alerts from Nagios to XMPP usernames.

http://www.gridpp.ac.uk/wiki/Nagios_jabber_notification

I have downloaded this, but have not yet got it working as of yet, but it does look promising.  It took me a while to update all the dependencies for it, but if those were in place, the installation itself is rather simple.  That is, the notification script works, but I haven't actually configured nagios as of yet.

This script has a shortcoming in that it only accepts the username portion of the JID and not the domain name -- and this means that notifications can only be sent between users of the same domain.

To illustrate, user@example.net can not send a message to user@example.com, but user1 and user2 on example.net can send to each other.

# this command will fail:
ytjohn@monitor:/usr/local/bin$ ./notify_by_jabber.pl yourtech\@example.com testing 
# while this one works
ytjohn@monitor:/usr/local/bin$ ./notify_by_jabber.pl yourtech testing


And in the perl script, you have to specify the login credentials and server you're connecting to:

## Configuration
# my $username = 'system@example.com';   # does not work
my $username = 'system';
my $password = "password";
my $resource = "nagios";
## End of configuration


my $len = scalar @ARGV;
if ($len ne 2) {
   die "Usage...\n $0 [jabberid] [message]\n";
}
my @field=split(/,/,$ARGV[0]);
#------------------------------------

# Google Talk & Jabber parameters :

my $hostname = 'talk.google.com';
my $port = 5222;
# componentname is the second half of your JID:
my $componentname = 'example.com';
my $connectiontype = 'tcpip';
my $tls = 1;

I am getting more and more into the idea of using XMPP for sending messages.  XMPP (also known as jabber) is a very open protocol.  Anyone can throw up a server, and by publishing a few DNS records, your server can interact with any other XMPP server out there (that is open).  For a popular example, anyone can throw
up their server with their domain and send messages to someone using Google Chat.
    
I have a customer that has their domain email hosted by Google apps for domains, which also means that each person has Google Talk, which is based off of XMPP.  For a monitoring scenario, I decided I'd rather have alerts go to an installed Google Talk client instead of to their email.  This would make alerts more noticeable, but less intrusive to their Inbox.  Google would also tie all their messages together (chat and email) for searching later. So my idea was to have monitoring server send the email alert to a local user, and have it piped through a program that would send the XMPP alert.  In the future, perhaps an XMPP bot could also accept commands to control the monitoring system.
   
I wanted to create a program called "jabblast.py" which would accept either standard input, or a command line argument, and send a message to a pre-defined list of recipients.

  echo "sent from stdin" | jabblast.py
  jabblast.py -m "sent from a command line argument"

Looking at the python and XMPP options out there, I came across three that looked promising. 

  # xmpppy
  # sleekxmpp
  # twisted (the "words" extension)

I was really expecting to find exactly what I wanted to do already written.  However, it seems anyone writing an XMPP script is creating a bot that runs continuously and would need modified for a one-time event.
   
Sleekxmpp was actually the easiest to understand and work with, however the project could be in danger of abandonment.   Twisted looks the most advanced, and I am drawn to it because it seems to offer the most versatility.  In my mind, I suspect that if I learn twisted for this, I can just keep using it for all the other stuff I want to do.  However, it might be "too much" for this simple of an operation.  xmpppy is the middle ground.  The project seems relatively active and popular. 

The problem with all three projects is a severe lack of documentation -- for twisted, this lack is primarily on the xmpp side.  One thing very well hidden is how to specify the server name you want to connect to.  Ideally, this is provided through a DNS record lookup, but that method eliminates experimentation with local network ip addresses.   Sleek never even approaches the concept that you might want your XMPP program to end.   It has a call for disconnecting, but when I call that, sleek immediately tries to reconnect.

I'm going to go ahead and create my program using all three methods and decide later which I like best.

For these examples, I have hardcoded the login credentials, as well as the "to" into the source code itself.  In the real world, I should be pulling that from a configuration file, either XML or YAML (and potentially from a remote web page).  Secondly, I only list one "to".  I believe that going from sending a single message to sending multiple messages should be a matter of a for loop.  Some libraries might even offer a feature to send to multiple recipients at once.
 
For my first example, I have created a script in sleekxmpp.  I wrote this by looking at the EchoBot example found here.  The script will send a message and then hang for a while after disconnecting before it finally exits.  I don't know why.  If you enable logging, an error is thrown after you pass in the disconnect command.  But otherwise, a relatively simple bit of code, easy to expand upon.

#!/usr/bin/env python

import sys
import logging
import sleekxmpp


# Uncomment the following line to turn on debugging
#logging.basicConfig(level=logging.DEBUG, format="%(levelname)-8s %(message)s")

def main() :
  # configure jabber id credentials and to
  to = 'recipient@example.net'
  myjid = 'sender@example.net/blastbot'
  mysecret = 'password'
 
  args = sys.argv[1:]
  if not args:
    message = sys.stdin.read()
  else:
    message = sys.argv[1]
 
  # print to, message
  # sys.exit()
 
  bot = BlastBot(myjid, mysecret, to, message)
  bot.run()
 

class BlastBot :

  def __init__(self, jid, password, to, message) :
    self.message = message
    self.to = to
    self.xmpp = sleekxmpp.ClientXMPP(jid, password)
    self.xmpp.add_event_handler("session_start", self.handleXMPPConnected)
    self.xmpp.add_event_handler("disconnected", self.handleXMPPDisconnected)
   
  def run(self) :
    self.xmpp.connect()
    self.xmpp.process(threaded=False)
   
   
  def handleXMPPDisconnected(self, event) :
    sys.exit()
   
  def handleXMPPConnected(self, event) :
    self.xmpp.sendMessage(self.to, self.message)
    self.xmpp.disconnect(self)
   
   
       
if __name__ == "__main__" :
  main()

On one of my customer's devices, the backup software requires that the backups go to a separate partition (or drive).  However, the customer only has one raid array and the bulk of the space is in /home.   To work around this limitation, I created a raw image file called backup.img, which gets mounted as /backup.   After the software performs its local backup, I use duplicity to backup /backup remotely to a backup server at my location (with encryption).

Today I got an alert that /backup was running low on space.  It was an 80GB image and 61GB was in use, leaving only 15GB free.  Now, this amount of free space should last quite a while.  However, the software (cPanel)  has a known issue for years that the 80% limit is hardcoded into the program.  I can change this, but every time cPanel updates, it overwrites that change.

So to be proactive, I decided to go ahead and increase the image size.

In order to increase the size of an image, you simply unmount your raw image and use the dd command.

# Increase by ~20GB
dd if=/dev/zero bs=1M count=20480 >> backup.img
# 20,480 is 20,480 MB or ~20GB

# check the filesystem
/sbin/e2fsck -f backup.img
# resize the filesystem
/sbin/resize2fs backup.img
# check the filesystem again
e2fsck -f backup.img               

I am reasonably familiar with Perl and PHP languages.  Recently though, I have taken it upon myself to learn Python.  I've played with Python a few times, and it seems to be very powerful with lots of useful functionality under the hood.

For some basic practice, I recommend Learn Python The Hard Way, which is free book by a reputable Python author, Zed Shaw.  He makes the book itself available for free as a PDF download or you can purchase it directly.  The book walks you through a number of examples, each one building upon what you learned in the previous exercise.  You are required to enter each exercise in by hand.  This uses the principle that if you learn better by writing things down, you'll learn better by typing it in.  Copying and pasting completely bypasses that manual entry learning process, so if you want to learn, I suggest you follow Shaw's instructions.  I found the book rather insightful, but I often felt that certain aspects were not being properly explained to me.  To be fair, I'm only up to exercise 15 in a 52 exercise book.

However, I also came across another way to learn Python.  Google has had instructor Nick Parlante teach Python in a classroom setting.  The class was video taped and put online along with the associated learning aids (primarily a group of python files).  The class was taught in 7 segments over 2 days.  So if you go to the class website, you can download the learning aids, and then follow along with the lecture videos. 

I like this method better than following the static book.  Since it was taught in an interactive classroom, it goes at a pace where you can keep up with it from your own computer.  Also, students in the class were able to ask questions, which allows the information to be represented in a different way.  Finally, at the end of each segment, there are some sample files you can open up in your editor.  Each file has a skeleton template.  In comments, it tells you what a segment is supposed to do, and you have to implement it in code based on what you learned while watching the video.  The sample file also includes some bare bones "unit tests" that call your functions, passing it input data and comparing it to an expected output.  These unit tests will show what they received from your function and what they expected.  If the two matched, you have succeeded. 

I like these practice samples mainly because it immediately forces you to apply the knowledge you learned.  You do have to think about it and do some back and forth troubleshooting on your own code.  Since they provided the unit tests, you can get instant feedback on if you're doing it correctly or not.

If you want to learn Python and can dedicate an hour on any given day to it, I recommend the Google course.  If you want to go in smaller 10-20 minute blocks, try using Learn Python The Hard Way.  Of course, you can pause the videos and come back to them, but I think it's much better to go through the entire video, following along on your own system as you go.

Starting around Memorial Day, we have been busy working on moving our office to our new location in Bedford, PA.  We have now completed that move and wanted to let everyone know our new address.

Our new address for correspondence is:

YourTech, LLC
300 Cumberland Rd
Bedford, PA 15522

A client wanted to make files available available to the web browser from within their LAN and a handful of static IPs without requiring any sort of username or password.  This is the web equivalent of a shared, read only folder.  This is no big issue, you can create an .htaccess file like so:

Order deny,allow
Deny from all
Allow from 10.10.0.1/16
Allow from 127.0.0.1/32
Allow from 1.1.1.1/32

However, they would also like to access it from a remote location with a username and password.

First, we need to create a password file.  In the old days, you would use "AuthType Basic", but a more secure method is the Digest method.  To use this, you must have the auth_digest module loaded into your Apache configuration.  If you are running a Debian or Ubuntu version, you can do this by executing "sudo a2enmod auth_digest".  Using digest authentication prevents your username and password from being sent in the clear (however, I always recommend that any site requiring authentication should utilize https).

Next,  you create your htdigest file:

htdigest -c .htdigest authname username

When prompted, you would enter a password for username.

Finally, you need to modify your .htaccess file to allow either method:

Order deny,allow
Deny from all
AuthName "authname"
AuthType Digest
AuthUserFile /var/www/.htpasswd
require valid-user
Allow from 10.10.0.1/16
Allow from 127.0.0.1/32
Allow from 1.1.1.1/32
Satisfy Any

Now, a user can come from the 10.10.x.x network, localhost, or 1.1.1.1 without requiring authentication.  If they later come from an unrecognized IP, they can enter their username and password and be granted access.

The Install

Make your gpg keys

The backup script

Essentially, what we want is a script that you just run and it will perform the backup for you. For my purposes, I want to backup to Amazon's Simple Storage Service (s3). To do this, you will need to sign up for the service (no cost to signup, just pay for space/bandwidth used) and get the AWS/AWS secret keys (think of them like username/passwords).

The following script will backup a directory called /mnt/backups to an Amazon bucket called j123backup (the bucket name must be unique between all Amazon S3 users). Please note that while you can use the same script and gpg keys on multiple servers (or have multiple backup scripts on the same server backing up different directories), you will want to make a separate bucket for each different source backup.

#!/bin/bash
# Export some ENV variables so you don't have to type anything
export AWS_ACCESS_KEY_ID=accesskey
export AWS_SECRET_ACCESS_KEY=secretkey
# GPG passphrase we used earlier
export PASSPHRASE=123456789012345

GPG_KEY=53F0891A

# The source of your backup
SOURCE=/mnt/backup

# The destination
# Note that the bucket need not exist
# but does need to be unique amongst all
# Amazon S3 users. So, choose wisely. 
DEST=s3+http://j123backup

# You can of course change your destination to an ftp or
# scp (ssh copy) server:
#DEST=scp://backupuser@backup.example.com/backups

duplicity \
    --encrypt-key=${GPG_KEY} \
    --sign-key=${GPG_KEY} \
    ${SOURCE} ${DEST} 

# this is an example of backing up multiple 
# directories at once and excluding others:
#
# duplicity \
#     --encrypt-key=${GPG_KEY} \
#     --sign-key=${GPG_KEY} \
#     --include=/home \
#     --include=/var/www/html \
#     --exclude=/var/www/html/cache/* \
#     ${SOURCE} ${DEST} 


# Reset the ENV variables. Don't need them sitting around
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export PASSPHRASE=

As a final note, you can definitely point your backup at another destination such as ftp or scp. I later ended up choosing an scp server over Amazon S3.

Yesterday I was working on a very old server -- a Cobalt RAQ 4.  I actually leased a RAQ 3 for web hosting back in 2001 and thought it was the neatest thing, but found it too limiting in many ways.  These servers were popular because they could be administered using a series of push buttons and an LCD screen up front.  It also has a web interface that controlled everything.  These servers were highly popular back in there day.

The Cobalt product line went away back in 2004 or so, but obviously a good number of these servers are still in use 5 years later.  As this server booted up, I saw an email address ending in @cobaltnet.com and wondered if that was still around.

Registrant:
Sun Microsystems, Inc.
   4150 Network Circle
   Santa Clara, CA 95054
   US

   Domain Name: COBALTNET.COM

   Administrative Contact, Technical Contact:
      Sun Microsystems, Inc.            hostmaster@sun.com
      4150 Network Circle
      Santa Clara, CA 95054
      US
      1-650-960-1300 fax: 650 336 6623


   Record expires on 15-Jun-2010.
   Record created on 30-Jul-2004.
   Database last updated on 30-Sep-2009 00:06:34 EDT.

   Domain servers in listed order:

   NS1.COBALT.COM
   NS2.COBALT.COM

So the cobaltnet.com domain is still in existance, at least until 2004.  However, it doesn't resolve to anything.  The reason for that is that the Cobalt.com domain name (notice ns1 and ns2.cobalt.com) itself has been purchased by another company, completely unrelated to the Cobalt server product.  So this is interesting in that Sun not only let cobalt.com go, but they never bothered to update the cobaltnet.com domain to point to an active name server. Sun paid about $2 billion for the Cobalt name and now it sits in a neglected corner of the Internet, just a few months away from finally expiring.

This is one of many such example found on the net of things that vanish without a trace. Cobalt was one of the first companies to really produce a polished interface for managing a web server.  Part of me wonders what would have happened if Cobalt/Sun had released that code as open source before the end came.  Would the community have picked it up and developed something amazing with it, or would it have vanished like the parent company.  Based on recent happenings with BeOS and Haiku, I suspect the former would have occurred.

I have been out of the programming circuit for a few years and have been looking at getting back into it.  My traditional programming style is an ssh window into my server and all my editing takes place on a development server, in vi.

Recently, I've been trying to decent work out a way to determine how the world sees your connectivity from within your network.  Essentially, I wanted to simulate accessing one of my locally connected machines from the Internet.  Typically, you have to subscribe to a third party service to perform this service for you.  Coincidentally, I have been reading up on Google App Engine and saw the potential in using GAE for my purpose.  I could envision writing a monitoring system that runs entirely on the GAE.  Unfortunately, I had no programming experience in Python or Java.  I did see that PHP has been ported to Java and someone got PHP running on GAE.  The possibility of either creating a new monitoring system in PHP (or modifying an existing PHP-based monitoring system) entered my mind.

I decided that rather than stay with PHP, I would use this project as a method to learn Python.  I started digging into Python resources and contemplating how I would want my monitoring system to work.  Ultimately though, I decided I didn't want to create a brand-new monitoring system (even a basic one) when existing ones such as Nagios and Zabbix do perfectly well. In my research, I found a project called mirrorrr that used GAE as a web-proxy. 

This solution was immediately obvious.  My existing monitoring system (Zabbix) has support for fetching web pages.  I could place a file with the word "OK" on my local web server and then fetch it through GAE.   I could even determine through the returned page whether my server was down or if GAE was down.

I set to work testing out the mirrorrr code under my own account.  The major issue I observed is that mirrorrr is configured to cache pages, meaning that when I changed my OK to FAIL, mirrorrr never updated the page.  In the closed-source world, that is the end of the story.  However, since this is an open-source project, this was an opportunity.

I've been wanting to get back into programming, and I when I start back, I want to be familiar with using an IDE (namely Eclipse).  In preparation for creating a monitoring system, I had setup a development station in (*gasp*) Windows Server 2003.  I connect to this via remote desktop and generally leave Eclipse running 24x7. I had also gone through the steps of installing the PyDev plugin, the GAE plugins, and SVN plugins.

The Process

I downloaded a copy of the code using SVN checkout and set to work editing the mirror.py file to disable CACHE.  I call this the "dive in and learn to swim later" process.  Here, I could make changes to existing code and test them out immediately.  In fact, the SDK for GAE works as a sort of mini-server.  Once I run the code within the SDK, any change I make to the source affects the running instance. 

I was able to read through and alter the code to allow me to switch between caching and non-caching.  I ran into an issue with their "recent urls" feature.  This shows the last 5 urls you have visited.  When you are not caching your data, this never gets sets and starts throwing errors.  I realized I would have to improve that section of the code before I could truly implement a configurable "enable cache" option.

At this point, I backed out of my file and considered my options.  I wanted to make two distinct changes to the source, one of which requires the other.  The author of the project hasn't maid a change to his(her?) code since December of 2008.  However, I did see recent entries in the Wiki, indicating this wasn't an abandoned project.  I realized that to truly make my changes worthwhile, I should try and get them included back in the upstream.  To do so, I should submit each change separately. That required more tracking than I had been doing.

So, I took care of another todo list item.  I went ahead and setup an "official" repository server for YourTech, checked out a fresh copy of mirrorrr, and then imported it into mine.  Now I could work.  I imported the project from my repo server into Eclipse and started recreating my work.  First I added added a feature to disable the recent urls.  At the same time, I made an improvement by moving a chunk of code into a self-contained method (which is apparently what python calls functions, as near I can tell at this juncture).  Once this was committed, I went ahead and proceeded with recreating my work on disabling the cache.

Once I was done, you could enable or disable each feature separately.  However, if you disabled the cache but left recent urls enabled, your recent urls would never update.  On the flip side, if recent urls were recorded before disabling the cache, then you could see the most recent urls before caching was disabled.  Some person may want that feature -- they could start up the mirrorrr, visit several links, then disable the cache, preserving those links on the main page forever.

At this point, the only remaining task was to submit my changes to the project maintainer.  I opened issues 6 & 7 and now I await response.

Conclusion

The more I use Eclipse, the more I like it.  The ability to perform every step of the process in one program is extremely useful.  Steps like comparing history or checking out a specific version is much easier to grasp than when you are tooling around the command line.  In the shell, I would typically have most of my files open as a background task.  Reverting a file from subversion required switching back to the file, closing it, then reverting.  In Eclipse, it's a right-click operation, regardless of whether the file is open or not.

Python's syntax is a bit weird coming from a Perl and PHP background, but it's learn-able.   As I plan to make several more improvements to mirrorrr, I hope to become proficient in this language as well.  However, I may be picking up a Perl project in the near future using the Mojo toolkit, so everything is up in the air.