postgresql hstore is easy to compare

hstore is an option key=>value column type that’s been around in postgresql for a long time. I was looking at it for a project where I want to compare “new data” to old, so I can approve it. There is a hstore-hstore option that compares two hstore collections and shows the differences.

In reality, an hstore column looks like text. It’s just in a format that postgresql understands.

Here, we have an existing record with some network information.

hs1=# select id, data::hstore from d1 where id = 3;
 id |                          data                          
----+--------------------------------------------------------
  3 | "ip"=>"192.168.219.2", "fqdn"=>"hollaback.example.com"
(1 row)

Let’s say I submitted a form with slightly changed network information. I can do a select statement to get the differences.

hs1=# select id, hstore('"ip"=>"192.168.219.2", "fqdn"=>"hollaback01.example.com"')-data from d1 where id =3;
 id |             ?column?              
----+-----------------------------------
  3 | "fqdn"=>"hollaback01.example.com"
(1 row)

This works just as well if we’re adding a new key.

hs1=# select id, hstore('"ip"=>"192.168.219.2", "fqdn"=>"hollaback01.example.com", "netmask"=>"255.255.255.0"')-data from d1 where id =3;
 id |                           ?column?                            
----+---------------------------------------------------------------
  3 | "fqdn"=>"hollaback01.example.com", "netmask"=>"255.255.255.0"
(1 row)

This information could be displayed on a confirmation page. Ideally, a proposed dataset would be placed somewhere, and a page could be rendered on the fly showing any changes an approval would create within the database.

Then we can update with the newly submitted form.

hs1=# update d1 set data = data || hstore('"ip"=>"192.168.219.2", "fqdn"=>"hollaback01.example.com", "netmask"=>"255.255.255.0"') where id = 3;
UPDATE 3

hs1=# select id, data::hstore from d1 where id = 3; id |                                         data                                         
----+--------------------------------------------------------------------------------------
  3 | "ip"=>"192.168.219.2", "fqdn"=>"hollaback01.example.com", "netmask"=>"255.255.255.0"
(1 row)

Note that if I wanted to delete a key instead of just setting it to NULL, that would be a separate operation.

update d1 SET data = delete(data, 'ip') where id = 3;
UPDATE 1

http://stormatics.com/howto-handle-key-value-data-in-postgresql-the-hstore-contrib/

Programming Uniden AMH-350 for APRS

This is a narrative post. If you want to see my python program that calculates out the diode matrix, skip to the end or click here,

I recently received this “Force Communications AMH-350” radio. Actually, it was an entire cabinet with a large power supply, an MFJ TNC2 tnc, and an old DOS PC running JNOS. These had active in a tower shed and turned off 3 years ago. The club wanted me to repurpose this packet system for APRS.

Once I plugged it in, the computer booted up to JNOS, but the radio and TNC did not turn on. The power supply had a plastic box on the back with a larger bussman 30A fuse. When I pulled it out, corrosion dust leaked out. I made a trip to the hardware store and replaced it. The radio turned on but not the TNC. On the front I found 3 smaller fuses and a note describing that “F3” ran the TNC. Pulled that fuse out and it was dead. A second trip to the hardware store got this fuse replaced. Then I plugged everything back in and turned on the power supply. Within 10 seconds, the “make it work smoke” had leaked out of the TNC2. This is probably why the F3 fuse had blown in the first place. This was disappointing, because there is new firmware for the TNC2 that makes it a decent APRS TNC, no computer needed.

The computer, I deemed too old to run a soundcard packet (using direwolf as my driver), so this left me with the power supply and radio. Grounding out the PTT line and using a frequency counter, it showed me “channel 2” was transmitting on 145.050. Channel 1 was not programmed at all.

A quick google search told me that Uniden bought Force Communications and sold this radio as a Uniden AMH-350. I found 2 other people looking for how to program it (one in 1994, and the other in 2004) with no response. I found someone selling the radio’s manual on ebay for $20. I offered them $10 and received the manual earlier this week.

The radio itself is programmed with a common cathode diode matrix, representing a binary value. Here is a picture of one back side of it programmed for 145.050. The manual provides a table covering frequencies from 148Mhz to 174Mhz in 5khz increments. Fortunately, it provides a formula on how to come up with your own frequencies. I ran through this formula multiple times getting different results from the book, till I realized the book was rounding some values UP or outright disregarding fractional parts. It also took a bit to wrap my head around binary “1” being disconnected (or cut) and binary “0” being connected. That felt backwards to me.

Eventually though, I was able to match the book, create a chart that matched the existing programmed 145.050 frequency (both Tx and Rx, which are programmed separately). Then, I wrapped the whole thing up in a set of python functions inside an ipython notebook. You can view this on ipython’s nbviewer or the direct gist.

I don’t have the radio programmed yet. I feel getting the diode matrixes out of “channel 2” and still having them useful for programming with is going to be difficult. I will need 7 diodes connected for each Tx and Rx slot, 14 total. I am attempting to program up channel 1. By the time I got to this portion, I was a bit tired and making mistakes, so I called it a night. Once I get to building out the programming board, I’ll post some more pictures.

rPI DPI Display, cheap.

Recently the internet noticed the Raspberry Pi could drive LCD panels using DPI. This allows very inexpensive displays to be used with basically no additional hardware.

This is not a full post, just capturing some details from someone elses blog post so I don’t lose it (will my site become my next bookmark holder?).

Let’s add a dirt cheap screen to the Raspberry Pi B+

Summary:

Total: $43.85

That being said, I see a Pi: 7″ Display no Touchscreen 1024×600 w/ Mini Driver for $70 with no messing with HDMI.

I also recently purchased a 5″ 800×480 screen with an hdmi driver off of a chinese vendor on ebay for $42.75. It just takes a month or so to arrive.

http://www.vslcd.com/Specification/VS-TY50-V2.pdf

Enter the Matrix

I used to be a big proponent of xmpp. However, over the years my enthusiasm has waned for it. I’m not the only one. Essentially, these days if your chat service is not done over HTTP(s), and if it doesn’t have persistence, your chat service is now legacy. Yes, I still enjoy IRC, and I think it’s great for ephemeral communications. But in this multi-device, mobile world, it’s hard to use IRC as a daily driver for my friends and coworkers.

Several months ago, I started looking into chat systems again for a different reason than most – amateur radio. There’s this thing in amateur radio called Broadband-Hamnet, which is a wireless mesh network. It’s not the first mesh system out there, but it has a really good initiative behind it. The idea behind it is that all nodes are configured to use the same SSID and the network is self configuring. If I stand up a node here at my house, someone else, having never spoken to me before, could deploy a node within range of mine and the two would connect. They would be able to see the node, any services I offer, and use them. DNS and service advertising is built in.

I wanted to come up with some “generic” mesh nodes with a connected server (raspberry pi). The idea that you could grab a couple of these boxes, deploy them in the field and operators would be able to share files, chat, and even video. The big catch was that you never knew what systems would be online at any given time.

I looked into standing up an IRC server with a web front end. This had a problem in that no historical messages would be synchronized during a netjoin. There are a number of P2P chat systems, though most of these require some sort of “bootstrap” system. Even worse, for an amateur radio system under FCC regulation, most of these are focused around encryption. Tox.im would be a good choice, except it would violate the no message obscuring rule of FCC part 97 that governs the Amateur Radio service.

I even started conceiving of a system based on the idea of a pub/sub message queue, except json over http. Nodes would subscribe to a channel and any message posted to a channel would get propagated to all the subscribing nodes. Using twisted, I could also create gateways for standard IRC or XMPP clients.

Well fortunately for me (and you) a group went out and did just that, only much much better than anything I could have put together. Matrix.org has put together a federated chat specification. The concept is really simple – json over http(s). They have a reference implementation called Synapse that is written in twisted. People run homeservers of synapse and will join channel. A channel is shared between all homeservers that subscribe to it and all channel events are propogated until consistency is achieved. This means that if a homeserver joins the channel late, or goes a way for a while, it will eventually achieve a complete history of all message events within the channel.

If you run your server on the default port of either 8008 for HTTP or 8448 of HTTPS, the only DNS record you need is an A record. If you use another port like 443, then you add a DNS SRV record stating the host and port (just like with XMPP).

While the project still has a few rough edges, it is definitely usable today. The most stable implementation is on matrix.org but you can also join my homeserver at matrix.ytnoc.net.

Everyday Superadmin

Every morning I wake up bleary eyed
Another victim of aggressive deadlines
I don’t think this is supportable
It’s not good at all

I’m just your average ordinary everyday super admin
Trying to save the build but never really done
I’m just your average ordinary everyday super admin
Nothing more than that, that’s all I really am

Just an all day job, there’s so much left to do
It’s kinda hard when everything fails on you
Try to make it look easy, gonna make it look good
Like anybody would

I’m just your average ordinary everyday super admin
Trying to save the build but never really done
I’m just your average ordinary everyday super admin
Nothing more than that, that’s all I really am

I’m just like everybody else
After all the hype it’s hard to tell
I keep my game face on so well

‘Cause I’m just your average, ordinary, everyday super admin
I’m trying to save the build
I’m just your average ordinary everyday super admin

I’m trying to find my token key
And no one knows it’s really me
It’s really me, it’s really me
Oh, yeah

I’m just your average ordinary everyday super admin
Trying to save the build but never really done
I’m just your average ordinary everyday super admin
Nothing more than that, that’s all I really am

I’m just your average ordinary everyday super admin
I’m trying to save the build
I’m just your average ordinary everyday super admin
Yeah, yeah

I’m just your average ordinary everyday super admin
Trying to save the build but never really done
I’m just your average ordinary everyday super admin
Nothing more than that, that’s all I really am

I’m just your average ordinary everyday super admin

All about git pull

All about git pull

Because you know
I’m all about git pull
’bout git pull, no copies
I’m all about git pull
’bout git pull, no copies
I’m all about git pull
’bout git pull, no copies
I’m all about git pull
’bout git pull

Yeah, it’s pretty clear, I ain’t no guru
But I can break it, break it
Like I always do.
And now I got that pull request, that we all want.
With all the right code in all the right places.

I see that manual push with rsync
We know that shit aint right
It caused my script to stop.
If you got fixes fixes, just check ’em in
So every line of code is peer reviewed
from the bottom to the top

Yeah, my coworker he told me don’t worry about the style
He says “We just need a little more testing to make it right.”
You know I won’t trust the logs to /dev/null
So if that’s where you put them then you better use syslog.

Because you know
I’m all about git pull
’bout git pull, no copies
I’m all about git pull
’bout git pull, no copies
I’m all about git pull
’bout git pull, no copies
I’m all about git pull
’bout git pull

Radio Musings

Completely unrelated to the post below, I discovered that some has forked a copy of my website on github. I’ve actually migrated from the static site to Mezzanine, but the repo is still in place. I was very surprised someone has forked it. I checked the user out and he recently forked a large number of repositories, all related to docker or coreos type things. It appears that “builtdock” is putting together some sort of platform as a service offering. I wonder if forking my site was an accident, or if they are planning to use it as a base of making their own static site.


I recently took my Raspberry Pi, connected a GPS, installed Xastir and have that setup as a sort of tactical display in my office. [](https://lh3.googleusercontent.com/–RicSliZ2kE/VDb9oB4auGI/AAAAAAAAT1M/eCkTaa8AIEg/w876-h1168-no/14%2B-%2B1 target=”_new”).

I have some interesting things I came across recently.

The first interesting project is from hackaday, a 0-30Mhz Portable SDR with transceive capabilities, opens-source design. It’d be interesting do this, but use a Pi+Touchscreen for the CPU portion.

http://hackaday.io/project/1538-portablesdr

The second one has been around for a while, and it’s called Earl. This is a crowd-funded, e-ink touchscreen android system with gps, built-in maps, and a VHF/UHF transceiver for GMRS/FRS/2m/70cm walkie talkie type capabilities. The radios are software defined, so something like DroidAPRS would be able to talk to the radio directly (perhaps with a bit of coding). I’ve been keeping and eye on this project (because I want one), but never thought it would actually materialize due to FCC rules concerning GMRS and FRS type radios. However they are now actively working through the FCC approval process and modifying the radio portions to get acceptance. I am hoping that enough get produced that some of these end up on ebay by non-hams that are disappointed in the radio communications aspect.

Python Modules and Single App OS

Emacs standing alone on a Linux Kernel. I’m not an emacs user, but I thought this was really fascinating. Basically, the author took the linux kernel, a statically compiled copy of emacs, the mount command, and that’s it. You could dd that to a hard drive and boot into only emacs. This gets me thinking about what I’m going to call SAOS – Single Application OS. Sort of like the python on a chip scene.

Python Apps the Right Way: entry points and scripts a very nice tutorial on making a python module, complete with setup.py. This one is very straightforward and simple. I’ll be referencing it and redoing some of the code I’ve worked on.

Salt The Earth

Learning Salt

I am just beginning to read up on SaltStack, but I am really liking it. It has a number of things I like from Ansible (separation from code and state), the targeting abilities of mcollective, and the centralized control of Puppet/Chef. Salt can be run masterless, but in a master/minion configuration, it uses a message queue (0mq) to control minions and get information back. All messages in this queue are encrypted using keys on both the minion and the master. If you distribute this key, you can consume salt generated data in other programs.

Running commands across all minions could look lik this:

salt '*' test.ping
salt '*' disk.percent

In these, test.ping and disk.percent are known as ‘execution modules’, which are essentially python modules that contain defined functions. For example, disk would be an ‘execution module’ and “percent” would be a function defined. Here, test.ping runs on all target hosts and returns “True”; disk.percent returns the percentage of disk usage on all minions.

You can also run ad-hoc commands.

salt '*' cmd.run 'ls -l /etc'

Of course, you can target blocks of systems by hostname (web* will match web, web1, web02, webserver,..). Salt also has something called Grains which everyone else calls facts. You can target based on the grains, or use salt to provide a report based on the grains. The following command will return the number of cpus for every 64-bit cpu.

salt -G 'cpuarch:x86_64' grains.item num_cpus

I think this would work as well:

salt -G 'cpuarch:x86_64 and num_cpus:4' test.ping

If not, these would work (Compound match):

salt -C '[email protected]:x86_64 and [email protected]_cpus:4' test.ping
salt -c '[email protected]:Ubuntu or [email protected]' test.ping

Salt States

In Puppet, your manifests and modules are very closely coupled in puppet code. In Ansible, they separate things into modules and “playbooks”. These playbooks are yaml files detailing what modules and values for ansible to use. Salt follows this pattern as well, separating execution modules from states with “Salt States”, aka SLS formulas (aka state modules).

A sample SLS formual for installing nginx would look like this:

nginx:
  pkg:
    - installed
  service:
    - running
    - require:
      - pkg: nginx

Assuming you save that in the right spot (/srv/salt/nginx/init.sls), you can apply this state module to all servers starting with the name ‘web’.

salt 'web*' state.sls nginx

So with one tool, you can query a large amount of data from all your minions, move them to a specific state, run ad-hoc style modules, etc. This can also all be expanded by writing your own python modules. Also, I’m mostly interested in the ability to target modules and groups of hosts in one command, but it’s worh noting that salt will do scheduling of jobs just like puppet and chef do.

Codebox in Docker

This is a quick note log. I was able to setup a self-hosted web IDE for programming. Today I tested out codebox.io,
but I also want to check out the open source Cloud9 IDE.

  • Codebox.io an open-source Web IDE for programming (in your browser)
  • Docker an open source container
  • Docker-Codebox. All I really want is this guy’s Dockerfile.

I already have Docker installed, that’s the easy bit.

Build the box:

git clone https://github.com/forty9ten/docker-codebox.git
cd docker-codebox
docker build -t ytjohncodebox .
# much building occurs

Run codebox in docker

This will run the codebox.io environment on port 8000 and you’ll be editing files that are stored in a directory
called workspace1. ~/workspace1 on the host gets mounted into /workspace1 in the container.

cd ~
mkdir workspace1
docker run -p 8000:8000 -v ./workspace1:/workspace1 -t ytjohncodebox -p 8000 run /workspace1
# many things happen
# I end up seeing this:
# Codebox is running at http://localhost:8000

Now, I can access my server on port 8000 (ie, http://192.168.1.32:8000/). This instance is unprotected, it just asks
for an email to get started. But the part I skipped over is that I am actually using nginx to proxy and password
protect this instance.

I can hit Ctrl+C on the terminal to cancel my instance and all my edited files are safely stored in ~/workspace1.

Interesting bits:
I create a new file in the web browser and save it, I see this bit of json in the console output:

[events] watch.change.create : { change: 'create',
  path: '/irule.txt',
  stats: 
   { current: 
      { dev: 64513,
        mode: 33188,
        nlink: 1,
        uid: 0,
        gid: 0,
        rdev: 0,
        blksize: 4096,
        ino: 262847,
        size: 27,
        blocks: 8,
        atime: Wed Feb 26 2014 05:18:44 GMT+0000 (UTC),
        mtime: Wed Feb 26 2014 05:18:42 GMT+0000 (UTC),
        ctime: Wed Feb 26 2014 05:18:42 GMT+0000 (UTC) },
     old: null } }

One thing I really need is the ability to open a web terminal. When I do though, the terminal window appears for a
few seconds, and then vanishes. In the console, I see this:

[log][shells.stream] new socket connected
[log][shells.stream] open shell  { shellId: 'term2020-44',
  opts: { rows: 80, columns: 24, id: 'term2020-44' } }
[log][events] shell.spawn : { shellId: 'term2020-44' }
[log][events] shell.attach : { shellId: 'term2020-44' }
[log][events] shell.open : { shellId: 'term2020-44' }
[log][events] shell.exit : { shellId: 'term2020-44' }
[log][shells.stream] socket disconnected
[log][shells.stream] socket disconnected
[log][hooks] use hook settings

I got this same issue going through my nginx proxy and connecting directly on port 8000. A bit of text flashes
quickly on the “terminal” in the web browser, but closes too quickly for me to catch it. Perhaps it’s attempting to
run something that is not installed in the Docker instance. If I can fix that, then I just need to come up with a cool
way to launch workspaces and tie them into my nginx setup (or switch over to hipache
as my front-end webserver).

Anyways, just wanted to record these steps here and show how easy it could be to get your own self hosted IDE.