I ordered the KS2 from http://www.ovh.com/ - $50/month, 3.4ghz, 16gb ram, and a 1TB software raid setup. My plan is to set this up as a single server virtual datacenter. They also have the SP2 with twice the ram and storage for $90, but I figured I'd test out the cheaper option first. I can always upgrade and migrate to a larger server later if I get into heavier usage. The prices are rather cheap and they have scripts that will automatically provision physical servers.
I had this installed with Centos 6. I tried first using the "ovh secure" kernel, but I could not get KVM working with that kernel, so I had it reinstalled with the "vendor" kernel. I allocated 20GB to "/" and the remainder (898GB) to "/data".
Installing kvm and libvirt is a simple yum command.
yum install kvm libvirt python-virtinst qemu-kvm
Then, on my workstation, I installed virt-manager, which allowed me to graphically create and install virtual machines (I can do this by hand on the server, but virt-manager is a nice way to get started). The connection is done over ssh, so it will either ask for your username/password, or it can use ssh-key authentication (preferred).
/data/vms to hold my installation isos and virual machines respectively. The trick I had to work out is that I couldn't just add "/data" as a directory-based storage volume, I had to make one for isos and one for vms. I also found that the default directory (/var/lib/libvirt/images) is rather difficult to remove. I disabled it and removed it, but it showed back up later. When creating through the dialog, virt-manager wants to put your vm image in "default".
Creating a new virtual machine using virt-manager and a downloaded ubuntu 12.04 iso image (in /data/isos) was rather slick. I created a new volume in /data/vms, set the memory and cpu and started it. The default networking is a NAT'd network in the 192.168.122.x/24 network. As ovh only provides 3 IP addresses for free, I'm content to start with this network for testing, but I plan to move to a different subnet mask.
If I need to nat ports, the libvirt site has a useful page on forwarding incoming connections.
iptables -t nat -A PREROUTING -p tcp --dport HOST_PORT -j DNAT --to $GUESTIP:$GUESTPORT iptables -I FORWARD -d $GUESTHOST/32 -p tcp -m state --state NEW -m tcp --dport $GUESTPORT -j ACCEPT
I have been reading some good things about The Formean, and how you can manage an infrastructure with it, so my next real VM will be an install of foreman. This will hopefully let me setup an enviroment where I can build virtual machines and provision them automatically. I don't know (yet) if The Foreman will handle iptable rules on the host, but it seems to have the ability to call external scripts and be customized, so I should be able to provision NAT on the host when provisioning a new VM.
Foreman utilizes DHCP and PXE to install "bare metal" VMs, so we need a network without DHCP. Now, to create my non-dhcp managed nat, I copy the default network xml file and modify it with my new range and remove the dhcp address
cd /usr/share/libvirt/networks cp default.xml netmanaged.xml
<network> <name>managednat</name> <bridge name="virbr1" /> <forward/> <ip address="172.16.25.1" netmask="255.255.255.0"> </ip> </network>
It should show up with
virsh net-list --all and I can activate it.
# virsh net-list --all Name State Autostart Persistent -------------------------------------------------- default active yes yes managednat inactive yes yes # virsh net-autostart managednat Network managednat marked as autostarted # virsh net-list --all Name State Autostart Persistent -------------------------------------------------- default active yes yes managednat inactive yes yes # virsh net-start managednat Network managednat started # virsh net-list --all Name State Autostart Persistent -------------------------------------------------- default active yes yes managednat active yes yes
The gateway will be 172.16.25.1, and I will assign the IP 172.16.25.5 to my Foreman virtual machine, aptly called "builder". Once the basic ubuntu machine is installed by hand (hopefully the last one we do in this environment), I'll want access to it. Ideally, this would be behind a firewall with vpn access, but I haven't got that far yet. So for now, I'll just setup some NAT for port 22 and 443.
iptables -t nat -A PREROUTING -p tcp --dport 8122 -j DNAT --to 172.16.25.5:22 iptables -I FORWARD -d 172.16.25.5/32 -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT iptables -t nat -A PREROUTING -p tcp --dport 8143 -j DNAT --to 172.16.25.5:443 iptables -I FORWARD -d 172.16.25.5/32 -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
Using "foreman-installer" is the most recommended method, ensuring that we have current packages directly from theforeman.org. I've installed 12.04 LTS (precise), so it's fairly straightforward, though I modified slightly from the installation documentation. The original instruction rely on running as root.
# get some apt support programs sudo apt-get install python-software-properties # add the deb.theforeman.org repository sudo bash -c "echo deb http://deb.theforeman.org/ precise stable > /etc/apt/sources.list.db/foreman.list" # add the key wget -q http://deb.theforeman.org/foreman.asc -O- | sudo apt-key add - # install the installer sudo apt-get update && sudo apt-get install foreman-installer # run the installer sudo ruby /usr/share/foreman-installer/generate_answers.rb
At this point, Foreman is running on port 443, or in my case "https://externalip:8143/". I can login with the username "admin" and the password "changeme".
I've been reading the manual more at this point, but I think my next step is to watch the video Foreman Quickstart: unattendend installation. If I can grok that (and it looks nicely step by step) I'll try and setup an unattended install.
There were a few hic-cups. Many of my older pages were not originally written in markdown, but rather using whatever CMS editor I had at the time. Pelican has an import feature, but some things got lost in translation. Any preformated code blocks had to be re-created manually (amusingly enough, copying and pasting from the old web page into vim, then indenting them properly).
The biggest loss is a search engine. A CMS like Wordpress or a hosted solution like Blogger stores your content in a database and provides a search interface to that content. With static pages, not so much. Most of the web gets around this using Google's CSE, which I am trying as well. As of yet, almost nothing appears to be indexing on this page. If this continues, I will setup a stanadlone search engine that simply indexes the pages on this site. (UPDATE: Google has a nice fresh index of the website, covering everything except this page I wrote yesterday.) I'm still researching possibilities on that. Ten years ago, ht://dig was the premier application to use, but development on that seems to have stopped in 2004. Sphider looks like a good product, but again, development stopped in 2007. Small search engine development seems to have dropped off significantly with the rise of Google's popularity.
A copy of the website content and theme can be tracked on github.
This weekend I decided to takle both learning Ruby and working with AWS via the Ruby API. Having only played with both of these in the past, this presents two learning challenges at once. However, from past projects, this is how I learn best. I am somewhat familiar with AWS terms and once made a script in Python to fire up an instance. This was before Amazon came out with their management console, so I imagine things have come a long way since then (hopefully easier). I also played with Ruby for a while, but didn't have a decent project for it. Having a project with goals will hopefully keep me on track and give me a way to measure my progress.
My goals for this project are as follows:
First off, I need to setup a ruby development environment. Since I have used PyCharm in the past, I will try JetBrain's RubyMine for my editor environment. After installing this, the first thing I learned is that rails is not installed. I could install using apt-get, but Jetbrains recommends using RVM. It looks like a nice way to manage different versions of Ruby, rails, and gems. I know when I have installed Ruby applications requiring gems, gem versions was always a source of concern. It is very easy to get mismatched gem versions in the wild.
RVM install locally to \~/.rvm on linux, which is nice - you don't mess up any system wide ruby installations and keep everything local to your development environment. After installation, I had to figure out a couple bits with rvm.
rvm install 1.9.2# installs ruby 1.9.2
rvm list# lists versions of ruby installed
rvm use 1.8.7# use ruby 1.8.7
First, your terminal has to be setup as a login shell. This tripped me up for a while until I changed the settings in my terminal emulator. terminator has this as checkmark option.
ytjohn@freshdesk:~$ rvm list rvm rubies ruby-1.8.7-p371 [ x86_64 ] ruby-1.9.2-p320 [ x86_64 ] # => - current # =* - current && default # * - default ytjohn@freshdesk:~$ rvm use 1.8.7 RVM is not a function, selecting rubies with 'rvm use ...' will not work. You need to change your terminal emulator preferences to allow login shell. Sometimes it is required to use `/bin/bash --login` as the command. Please visit https://rvm.io/integration/gnome-terminal/ for a example.
After switching to login
ytjohn@freshdesk:~$ rvm use 1.8.7 Using /home/ytjohn/.rvm/gems/ruby-1.8.7-p371
Finally, once you get ruby and rails working, you can create your rails project. I'm starting with a rails project because it's "all the rage" and gives you a decent running start. Later, I'll work on switching the supplied templates with boilerplate + bootstrap based ones.
This gets me started. Next up, I'll actually create the project from within RubyMine and just work on basic web functionality.
Yesterday, I ran into a small, but confusing issue converting from markdown to html for a post. A markdown process will convert html in a code block to html escaped entities. That way, when you use the resulting html, your html example code doesn't get interpreted as html.
<p>This is a <strong>strong</strong> example.</p>
Gets converted to:
<pre><code><p>This is a <strong>strong</strong> example.</p> </code></pre>
In the "dingus" I made, this didn't appear to be happening. The above example rendered as:
<pre><code>This is a <strong>strong</strong> example.</p></code></pre>
Furthermore, it worked perfectly fine using the "official" php-markdown dingus. I was using their library, and it's incredibly simple to implement. After some digging, I discovered that in my dingus, the code was being converted properly in my preview section, but not in my HTML Source textarea. I was printing the same
$render variable in both sections, but getting different results in my browser.
As it turns out, most html elements are "CDATA" and a textarea is "PCDATA". When all is said and done, this means that instead of needing to send <, I need to send < to the browser. Fortunately, php has a function called htmlspecialchars() that does this for me. For my HTML Output, I just needed to change
print $render to