Enabling remote logging

I was experimenting with rsyslog to allow remote logging and within 20 minutes, I had two rogue log entries show up from random hosts.

Jan 17 05:00:23 64-181-24-18.unassigned.ntelos.net kernel: Configuring keys for mac:14:5a:05:ad:6b:cd, sc_hasclrkey is set
Jan 17 05:01:05 64-181-24-18.unassigned.ntelos.net kernel: Configuring keys for mac:14:5a:05:ad:6b:cd, sc_hasclrkey is set

So what is interesting about this? I'm on comcast in PA. The host that sent this is on ntelos.net, or Lumos in (most likely) West Virginia. For some reason, a system there is sending a log message to my IP address. I just now am accepting it.

Audible Thoughts

Recently, I've been working my way through the Ender's Saga audio books. I had read Ender's Game back in grade school, but I didn't know there was a whole series. They're a really good read.. or listen, as the case may be.

I've been listening to the Ender series in mp3 format. First, I had them on a thumb drive which was read by my car's radio. I find it very convenient that a radio in my car can directly read a thumb drive's file-system and play any audio files found on it. My wife has the same feature in her car and updates her thumb drive quite often with different music.

When I was listening to Ender's Game, I noticed that sometimes when I turned off the car, it would forget my place in the current file and start from the beginning. With Ender's Game, the book was split up into lots of small MP3s, several per chapter, so this wasn't too big a deal. I soon determined that the trick was to make sure the track was playing before I shut the car off. If I had it paused for some reason (say a phone call) when I shut the car off, the file would restart when I started the car back off. If I just turned off the car while it was playing, it would resume perfectly.

However, after listening to the third book, "Xenocide" for around a week, somehow it went back to the beginning of the book. I did some fiddling with the and found I couldn't find my place. After further examination, I discovered that this book is split into 3 MP3s, each over 6 hours long. My radio doesn't have the ability to forward within each track. Not many MP3 players do.

I was able to resolve this by moving the book to my Android phone and setting up a Winamp playlist. Here, I can slide a little dot to any position in the file (and therefore the story) I want. I can also listen to the book outside of the car and keep my place. Rather nice.

I am still bit annoyed about the need to do this. To me, this represented a huge shortcoming in audio books. After reflecting on this, I realised it wasn't a shortcoming in audio books, but rather in the particular format and players involved.

In my last vehicle, I had listened to some audio books on mp3 on cd. Here, it was one huge mp3 for the entire book. However, the cd player always resumed at the same place. I knew that if I took the CD out, I would go back to the beginning though. This was another shortcoming. The CD player considered each MP3 a track and couldn't fastforward within a track itself, only from track to track.

Some of the original audio books were on tape. I suppose there might have been some on records, but the first format I've ever been familiar with was cassette tape. Tape players had no concept of tracks, everything was just a position on the tape. Ok, I know I had one tape player that would attempt to detect when one song ended and the other began, which would stop a fast forward when it reached a silent spot. But by and large, you navigated every second of the tape using fast forward and reverse. Tape was probably the best implementation of audio books.

Then came audio books on CD. Most CD players I knew back then, along with some modern ones we have in our house now works track by track. Discmans, home theatre stereos and vehicle cd players almost universally have this limitation. I know a few players that can fast forward within a track, but there isn't that big of a market for this feature, so it gets left out. Audio books on CD were a step backwards from tape.

Today, we have audio books in a digital format such as mp3 or wav. These have a big advantage that they can be purchased online and downloaded. But unless you have a "smart" player such as a computer, ipod touch or smartphone, you have the same limitation as a CD. Even the non-touch ipods have 'track by track, file by file' controls.

I suppose this problem will correct itself as tablets, smartphones, and all manner of personal computing devices proliferate. For now though, a decent audio book experience is limited to tape or "smart players". There is a wide range of inexpensive (and expensive) mp3 players that will most likely end up frustrating anyone getting into audio books.

Impressive Notebook

Those of you that know me probably wouldn't guess this, but I actually work with a bunch of nerds, geeks, and freaks. Pretty much all of my coworkers have smartphones, and many of them have at least one tablet. Every time a new portable computing device launches, it's a guarantee that someone in my group will show up with it. Since all these devices are nearly similar, no one is really impressed when a new one shows up in the office.

A couple weeks ago, a friend of mine was studying for the GMAT test. With the GMAT materials is several sheets of laminated graph bound together. The idea is that you can practice on this with dry-erase markers while studying. I thought this was an ingenious idea. I had once tried to use a small whiteboard for temporary notes, but it lost its usefulness after a few notes. The graph paper "notebook" worked out to about 6 long sheets (3 pages front and back, legal sized), so you had plenty of room to take notes. I looked them up online, and one pad was running around $25. I tabled that idea.

This week, I remembered the pad and thought "I could just get some graph paper and laminate it myself!". I stopped in a Big Lots and Dollar General. No graph paper. Then I went to target and found some. But while I was looking at that and trying to decide how I would bind it, I came across a  5-Star Flex Binder. The binder itself could be opened flat and came with dividers, graph paper, and regular lined paper for $10. I figured this would work out better than any homemade solution I would come up with.

Anyways, I carried this thing fresh from the store into work. It caught everyone's attention. In the parking lot, a coworker was like "look at you all impressive with your notebook". I had it lying on my desk and everyone that passed it took notice. Where a $500 tablet gets a yawn, my $10 notebook was drawing a surprising amount of interest. I may have started something new.

To complete my story, I discovered that I did NOT have to laminate the graph paper. Three of the "divider pages had an insert for a full sheet of paper. I was able to place the graph paper into these plastic sheathes and they work great for dry erase markers.

 

 

It was a  so I can definitely see why they are envious. Even the binder rings are flexible!

For Sale: Dirt Devil Ultra Vision Turbo.

For Sale:  Dirt Devil Ultra Vision Turbo.
Price: Make an offer
"It Kinda Sucks" - John Hogenmiller, USMC

I once owned a regular Bissel bagged vacuum cleaner.  For the better part of a decade, I would turn it on and vacuum the floor. Before that, my sister owned the vacuum and did pretty much the same thing.  It was a rather boring device that picked up debris from the carpet and would place it in a bag.

Like all good things, this boring vacuum cleaner broke.  A plastic part that held the upright portion to the base developed a crack.  This prevented the belt from tightening properly, leading to breakages.  Perhaps some JB weld would have held this piece together.  However, we had received some money from our wedding and decided to replace this 10+ year old appliance.

We picked up a Dirt Devil Ultra Vision Turbo.  Robin had been saying she wanted something red, shiny, powerful, and fast.  I felt this "turbo" vacuum cleaner would fit the bill.  As we began using it, I often thought back to how boring the old Bisell was compared to this new vacuum.  You see, Dirt Devil had done some research and found that most people found vacuuming to be a boring job.  In a survey, 93% of all respondents stated that they would rather do "anything else" than vacuum.  Dirt Devil decided to change all this.  They had all the data.  People hated vacuuming.  The Ultra Vision Turbo was designed: not so much with vacuuming in mind - but to challenge people.
    
That's right.  Before getting this vacuum, it had never occured to fight with my vacuum.  It never occured to me to question its functionality.  Dirt Devil has changed the way I look at vacuuming.  The first thing I noticed when we first used it was that it seemed to only pick up half the dirt.  In fact, I often had to question if it was picking up the dirt or if it was just spreading it around.  I knew that if I turned it off in one room and carried it to another, dirt from the first room would be deposited in the second.  This was part of the mental challenge.

The next thing I noticed was that it was constantly getting clogged.  With the older bagged vacuums, air flowed through a 1" x 3" straight plastic chamber to the bag.  With the new bag-less (and even most of the bagged) vacuums, this is replaced with a 1.5" diameter bendy hose.  This feature isn't specific to Dirt Devil, but it does an amazing job of trapping dirt, lint, and dog hair in each bend.  The cool thing that Dirt Devil did is that they require you to use a philips screwdriver to remove the hose enough to unclog it.  You can plan on doing this at least once for every two times you vacuum.

The other nifty feature of this vacuum is a thin, fragile belt.  The thin belt is crucial.  If you had a durable belt like the older vacuums did, you might never have to replace it.  Our last vacuum had the same belt for over 10 years -- the original belt from the store.   With the Dirt Devil, you get to replace the belt once a year or more.  The vacuum currently needs its third belt in under two years.  Much like the last time I went to pick up one of these belts from the store, Wal-Mart is sold out.  They are a hot commodity.  They have the same exact vacuum, some filters, and an empty peg where the belts would go.  It's a lot like new game consoles being sold out everywhere, but for all eternity.
   
Sometimes.. you just don't know what the problem is.  In this case, you do exploratory maintenance.  This involves taking the vacuum apart and cursing at it until it works again.

So if you're looking for a vacuum that needs maintenance as much as your floor needs vacuumed, the Dirt Devil Ultra Vision Turbo is for you.   I picked up a Bissel bag-less.  I would have got the bagged version, but I saw it had the same hose setup as the bagless.  It did get the clogged hose, but the hose is transparent.  I can see the clog.  It lets me stretch the hose out to get the clog moving again.  It also picks up all the dirt I can see or feel with my bare feet.  It fails to dump dirt from one room into another.  It is.. a boring vacuum.

Sending messages using xmpppy

In continuing working with XMPP and Python, I managed to get xmpppy
working.

This proved to be particularly tricky because xmpppy uses some outdated
modules like md5 and sha.  It also relies on some dns functions which no
longer seem to work.

I was able to use the sample script "xsend.py" that the project
provides.  The big thing is that I had to specify talk.google.com and
the port number directly in the connect command.

If I did not specify the servername, I would get the error:
 An error occurred while looking up _xmpp-client._tcp.example.com

NOTE: DNS is configured correctly for the domain I was practicing with,
but the xmpppy code was not able to resolve it.

To run the script, simply execute;

./xblast.py [email protected]  text to send

The Script:

!/usr/bin/python

$Id: xsend.py,v 1.8 2006/10/06 12:30:42 normanr Exp $

import sys,os,xmpp,time

if len(sys.argv) < 2:
    print "Syntax: xsend JID text"
    sys.exit(0)

tojid=sys.argv[1]
text=' '.join(sys.argv[2:])

jidparams={}

jidparams['jid']='[email protected]'
jidparams['password'] = '123456'

jid=xmpp.protocol.JID(jidparams['jid'])
cl=xmpp.Client(jid.getDomain(),debug=True)

con=cl.connect(('talk.google.com',5222),  use_srv=False)

if not con:
    print 'could not connect!'
    sys.exit()
print 'connected with',con
auth=cl.auth(jid.getNode(),jidparams['password'],resource=jid.getResource())
if not auth:
    print 'could not authenticate!'
    sys.exit()
print 'authenticated using',auth

cl.SendInitPresence(requestRoster=0)   # you may need to uncomment

this for old server
id=cl.send(xmpp.protocol.Message(tojid,text))
print 'sent message with id',id

time.sleep(1)   # some older servers will not send the message if you
disconnect immediately after sending

cl.disconnect()


Nagios and XMPP

I found that someone has written a perl script geared towards sending
alerts from Nagios to XMPP usernames.

http://www.gridpp.ac.uk/wiki/Nagios_jabber_notification

I have downloaded this, but have not yet got it working as of yet, but
it does look promising.  It took me a while to update all the
dependencies for it, but if those were in place, the installation itself
is rather simple.  That is, the notification script works, but I haven't
actually configured nagios as of yet.

This script has a shortcoming in that it only accepts the username
portion of the JID and not the domain name -- and this means that
notifications can only be sent between users of the same domain.

To illustrate, [email protected] can not send a message to
[email protected], but user1 and user2 on example.net can send to each
other.

# this command will fail:
ytjohn@monitor:/usr/local/bin$ ./notify_by_jabber.pl
[email protected] testing 
# while this one works
ytjohn@monitor:/usr/local/bin$ ./notify_by_jabber.pl yourtech
testing


And in the perl script, you have to specify the login credentials and
server you're connecting to:

## Configuration
# my $username = '[email protected]';   # does not work
my $username = 'system';
my $password = "password";
my $resource = "nagios";
## End of configuration




my $len = scalar @ARGV;
if ($len ne 2) {
   die "Usage...n $0 [jabberid] [message]n";
}
my @field=split(/,/,$ARGV[0]);
#------------------------------------




# Google Talk & Jabber parameters :

my $hostname = 'talk.google.com';
my $port = 5222;
# componentname is the second half of your JID:
my $componentname = 'example.com';
my $connectiontype = 'tcpip';
my $tls = 1;




Jabbering about Python

I am getting more and more into the idea of using XMPP for sending
messages.  XMPP (also known as jabber) is a very open protocol.  Anyone
can throw up a server, and by publishing a few DNS records, your server
can interact with any other XMPP server out there (that is open).  For a
popular example, anyone can throw up their server with their domain and
send messages to someone using Google Chat.

I have a customer that has their domain email hosted by Google apps for
domains, which also means that each person has Google Talk, which is
based off of XMPP.  For a monitoring scenario, I decided I'd rather have
alerts go to an installed Google Talk client instead of to their email. 
This would make alerts more noticeable, but less intrusive to their
Inbox.  Google would also tie all their messages together (chat and
email) for searching later. So my idea was to have monitoring server
send the email alert to a local user, and have it piped through a
program that would send the XMPP alert.  In the future, perhaps an XMPP
bot could also accept commands to control the monitoring system.

I wanted to create a program called "jabblast.py" which would accept
either standard input, or a command line argument, and send a message to
a pre-defined list of recipients.



echo "sent from stdin" | jabblast.py
  jabblast.py -m "sent from a command line argument"

Looking at the python and XMPP options out there, I came across three
that looked promising. 

# xmpppy
  # sleekxmpp
  # twisted (the "words" extension)

I was really expecting to find exactly what I wanted to do already
written.  However, it seems anyone writing an XMPP script is creating a
bot that runs continuously and would need modified for a one-time
event.

Sleekxmpp was actually the easiest to understand and work with, however
the project could be in danger of abandonment.   Twisted looks the most
advanced, and I am drawn to it because it seems to offer the most
versatility.  In my mind, I suspect that if I learn twisted for this, I
can just keep using it for all the other stuff I want to do.  However,
it might be "too much" for this simple of an operation.  xmpppy is the
middle ground.  The project seems relatively active and popular. 

The problem with all three projects is a severe lack of documentation --
for twisted, this lack is primarily on the xmpp side.  One thing very
well hidden is how to specify the server name you want to connect to. 
Ideally, this is provided through a DNS record lookup, but that method
eliminates experimentation with local network ip addresses.   Sleek
never even approaches the concept that you might want your XMPP program
to end.   It has a call for disconnecting, but when I call that, sleek
immediately tries to reconnect.

I'm going to go ahead and create my program using all three methods and
decide later which I like best.

For these examples, I have hardcoded the login credentials, as well as
the "to" into the source code itself.  In the real world, I should be
pulling that from a configuration file, either XML or YAML (and
potentially from a remote web page).  Secondly, I only list one "to".  I
believe that going from sending a single message to sending multiple
messages should be a matter of a for loop.  Some libraries might even
offer a feature to send to multiple recipients at once.

For my first example, I have created a script in sleekxmpp.  I wrote
this by looking at the EchoBot example found here.  The script will
send a message and then hang for a while after disconnecting before it
finally exits.  I don't know why.  If you enable logging, an error is
thrown after you pass in the disconnect command.  But otherwise, a
relatively simple bit of code, easy to expand upon.

#!/usr/bin/env python

import sys
import logging
import sleekxmpp

# Uncomment the following line to turn on debugging
#logging.basicConfig(level=logging.DEBUG, format="%(levelname)-8s
%(message)s")

def main() :
  # configure jabber id credentials and to
  to = '[email protected]'
  myjid = '[email protected]/blastbot'
  mysecret = 'password'



args = sys.argv[1:]
  if not args:
    message = sys.stdin.read()
  else:
    message = sys.argv[1]



# print to, message
  # sys.exit()

bot = BlastBot(myjid, mysecret, to, message)
  bot.run()

class BlastBot :

def __init__(self, jid, password, to, message) :
    self.message = message
    self.to = to
    self.xmpp = sleekxmpp.ClientXMPP(jid, password)
    self.xmpp.add_event_handler("session_start",
self.handleXMPPConnected)
    self.xmpp.add_event_handler("disconnected",
self.handleXMPPDisconnected)




def run(self) :
    self.xmpp.connect()
    self.xmpp.process(threaded=False)

def handleXMPPDisconnected(self, event) :
    sys.exit()

def handleXMPPConnected(self, event) :
    self.xmpp.sendMessage(self.to, self.message)
    self.xmpp.disconnect(self)

if __name__ == "__main__" :
  main()

Expanding a raw image file

On one of my customer's devices, the backup software requires that the
backups go to a separate partition (or drive).  However, the customer
only has one raid array and the bulk of the space is in /home.   To work
around this limitation, I created a raw image file called backup.img,
which gets mounted as /backup.   After the software performs its local
backup, I use duplicity to backup /backup remotely to a backup server at
my location (with encryption).

Today I got an alert that /backup was running low on space.  It was an
80GB image and 61GB was in use, leaving only 15GB free.  Now, this
amount of free space should last quite a while.  However, the software
(cPanel)  has a known issue for years that the 80% limit is hardcoded
into the program.  I can change this, but every time cPanel updates, it
overwrites that change.

So to be proactive, I decided to go ahead and increase the image size.

In order to increase the size of an image, you simply unmount your raw
image and use the dd command.

# Increase by ~20GB
dd if=/dev/zero bs=1M count=20480 >> backup.img
# 20,480 is 20,480 MB or ~20GB

# check the filesystem
/sbin/e2fsck -f backup.img
# resize the filesystem
/sbin/resize2fs backup.img
# check the filesystem again
e2fsck -f backup.img               




Learning Python

I am reasonably familiar with Perl and PHP languages.  Recently though,
I have taken it upon myself to learn Python.  I've played with Python a
few times, and it seems to be very powerful with lots of useful
functionality under the hood.

For some basic practice, I recommend Learn Python The Hard Way,
which is free book by a reputable Python author, Zed Shaw.  He makes the
book itself available for free as a PDF download or you can purchase it
directly.  The book walks you through a number of examples, each one
building upon what you learned in the previous exercise.  You are
required to enter each exercise in by hand.  This uses the principle
that if you learn better by writing things down, you'll learn better by
typing it in.  Copying and pasting completely bypasses that manual entry
learning process, so if you want to learn, I suggest you follow Shaw's
instructions.  I found the book rather insightful, but I often felt that
certain aspects were not being properly explained to me.  To be fair,
I'm only up to exercise 15 in a 52 exercise book.

However, I also came across another way to learn Python.  Google has had
instructor Nick Parlante teach Python in a classroom setting.  The
class was video taped and put online along with the associated
learning aids (primarily a group of python files).  The class was taught
in 7 segments over 2 days.  So if you go to the class website, you can download the learning aids, and then follow along with
the lecture videos. 

I like this method better than following the static book.  Since it was
taught in an interactive classroom, it goes at a pace where you can keep
up with it from your own computer.  Also, students in the class were
able to ask questions, which allows the information to be represented in
a different way.  Finally, at the end of each segment, there are some
sample files you can open up in your editor.  Each file has a skeleton
template.  In comments, it tells you what a segment is supposed to do,
and you have to implement it in code based on what you learned while
watching the video.  The sample file also includes some bare bones "unit
tests" that call your functions, passing it input data and comparing it
to an expected output.  These unit tests will show what they received
from your function and what they expected.  If the two matched, you have
succeeded. 

I like these practice samples mainly because it immediately forces you
to apply the knowledge you learned.  You do have to think about it and
do some back and forth troubleshooting on your own code.  Since they
provided the unit tests, you can get instant feedback on if you're doing
it correctly or not.

If you want to learn Python and can dedicate an hour on any given day to
it, I recommend the Google course.  If you want to go in smaller 10-20
minute blocks, try using Learn Python The Hard Way.  Of course, you can
pause the videos and come back to them, but I think it's much better to
go through the entire video, following along on your own system as you
go.

We have moved!

Starting around Memorial Day, we have been busy working on moving our
office to our new location in Bedford, PA.  We have now completed that
move and wanted to let everyone know our new address.

Our new address for correspondence is:

YourTech, LLC
300 Cumberland Rd
Bedford, PA 15522