Vault Standup

This is a little walkthrough of settng up a “production-like” vault server with etcd backend (Not really production, no TLS and one person with all the keys). Hashicorp Vault is incredibly easy to setup. Going through the dev walkthrough is pretty easy, but when you want to get a little more advanced, you start getting bounced around the documentation. So these are my notes of setting up a vault server with an etcd backend and a few policies/tokens for access. Consider this part 1, and in “part 2”, I’ll setup an ldap backend.

Q: Why etcd instead of consul?
A: Most of the places I know that run consul, run it across multiple datacenters, and a few thousand servers, and interacts with lots of different services. Even if the secrets are protected, the metadata is quite visible. I want a rather compact and isolated backend for my eventual cluster.

Let’s get started.

First off, create a configuration file for vault.

vaultserver.hcl:

[email protected]:~$ cat vaultserver.hcl
storage "etcd" {
  address  = "http://localhost:2379"
  etcd_api = "v2"
  path = "corevault"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}

disable_mlock = true
cluster_name = "corevault"

Start the server (in its own terminal)

[email protected]:~$ vault server -config=vaultserver.hcl
==> Vault server configuration:

                     Cgo: disabled
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", tls: "disabled")

Init the server

dfzmbp:~ ytjohn$ export VAULT_ADDR=http://vaultcore01.pool.lab.ytnoc.net:8200
dfzmbp:~ ytjohn$ vault init
Unseal Key 1: f9XJwuxla/H86t8pbWVPnI6Tfi3nQtkasq303Oi8B+ep
Unseal Key 2: jFqEmE1c/lei+C1aIju6JM2t5fSI534g26E7Nv83t9RV
Unseal Key 3: ty/P+Jubm1BukPcdZ16eJFD0JQ9BFGqOSgft35/fvHXr
Unseal Key 4: 6k4aPjuKgz0UNe+hTVAOKUzrIvbS9w8UszB0HX3Au496
Unseal Key 5: PYNjRe9vBvHAGE9peiotrtjoYuVlAV/9QJ0NvqZScd2a
Initial Root Token: b6eac78d-f278-4d32-6894-a8168d055340

That Initial Root Token is your only means of accessing the vault once it’s unsealed. Don’t lose it until you replace it.

And this creates a directory in etcd (or consul)

[email protected]:~$ etcdctl ls
/test1
/corevault
[email protected]:~$ etcdctl ls /corevault
/corevault/sys
/corevault/core

Unseal it:

dfzmbp:~ ytjohn$ vault unseal
Key (will be hidden):
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 1
Unseal Nonce: d860cb16-f084-925d-6f41-d80ef15e297c
dfzmbp:~ ytjohn$ vault unseal
Key (will be hidden):
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 2
Unseal Nonce: d860cb16-f084-925d-6f41-d80ef15e297c
dfzmbp:~ ytjohn$ vault unseal
Key (will be hidden):
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce:
dfzmbp:~ ytjohn$ vault unseal
Vault is already unsealed.

Now let’s take that root token and save it in our home directory. Not safe, because it’s the all-powerful root token, you shold create a user token for yourself. But that’s later.

Save your token (or export it as VAULT_TOKEN), then write and read some secrets.

echo b6eac78d-f278-4d32-6894-a8168d055340 > ~/.vault-token
dfzmbp:~ ytjohn$ vault read secret/hello
Key                 Value
---                 -----
refresh_interval    768h0m0s
value               world

dfzmbp:~ ytjohn$ vault read -format=json secret/hello
{
    "request_id": "a4b199e7-ff7c-e249-2944-17424bf1f05c",
    "lease_id": "",
    "lease_duration": 2764800,
    "renewable": false,
    "data": {
        "value": "world"
    },
    "warnings": null
}

dfzmbp:~ ytjohn$ helloworld=`vault read -field=value secret/hello`
dfzmbp:~ ytjohn$ echo $helloworld
world

Ok, that’s the basics of getting vault up and running. Now we want to get more users to access it. What I want is to create three “users” and give them each a path.

infra admins = able to create, read, and write to secret/infra/*
infra compute = work within the secret/infra/compute area.
infra network = work within the secret/infra/network area

infraadmin.hcl

path "secret/infra/*" {
  capabilities = ["create"]
}

path "auth/token/lookup-self" {
  capabilities = ["read"]
}

infracompute.hcl

path "secret/infra/compute/*" {
  capabilities = ["create"]
}

path "auth/token/lookup-self" {
  capabilities = ["read"]
}

infranetwork.hcl

path "secret/infra/network/*" {
  capabilities = ["create"]
}

path "secret/infra/compute/obm/*" {
  capabilities = ["read"]
}

path "auth/token/lookup-self" {
  capabilities = ["read"]
}

Now, we write these policies in.

dfzmbp:vault ytjohn$ vault policy-write infraadmin infraadmin.hcl
Policy 'infraadmin' written.
dfzmbp:vault ytjohn$ vault policy-write infracompute infracompute.hcl
Policy 'infracompute' written.
dfzmbp:vault ytjohn$ vault policy-write infranetwork infranetwork.hcl
Policy 'infranetwork' written.

Let’s create a token “user” for each policy.

dfzmbp:vault ytjohn$ vault token-create -policy="infraadmin"
Key             Value
---             -----
token           d16dd3dc-cd9e-15e1-8e41-fef4168a429e
token_accessor  50a1162f-58a2-474c-466d-ec68fac9a2f9
token_duration  768h0m0s
token_renewable true
token_policies  [default infraadmin]

dfzmbp:vault ytjohn$ vault token-create -policy="infracompute"
Key             Value
---             -----
token           d156326d-1ee6-7a93-d9d3-428e2211962d
token_accessor  daf3beb4-6c31-4115-2d00-ba811c50b05b
token_duration  768h0m0s
token_renewable true
token_policies  [default infracompute]

dfzmbp:vault ytjohn$ vault token-create -policy="infranetwork"
Key             Value
---             -----
token           84faa448-20d9-b472-349f-1053c81ff4c9
token_accessor  68eea7ec-78c0-4be1-03c4-f2ec155b66de
token_duration  768h0m0s
token_renewable true
token_policies  [default infranetwork]

Let’s login as with the infranetwork token and attempt to write to compute. I have not yet created secret/infra/compute or secret/infra/network and I’m curious if infraadmin is needed to make those first.

dfzmbp:vault ytjohn$ vault auth 84faa448-20d9-b472-349f-1053c81ff4c9
Successfully authenticated! You are now logged in.
token: 84faa448-20d9-b472-349f-1053c81ff4c9
token_duration: 2764764
token_policies: [default infranetwork]
dfzmbp:vault ytjohn$ vault write secret/infra/compute/notallowed try=wemust
Error writing data to secret/infra/compute/notallowed: Error making API request.

URL: PUT http://vaultcore01.pool.lab.ytnoc.net:8200/v1/secret/infra/compute/notallowed
Code: 403. Errors:

* permission denied
dfzmbp:vault ytjohn$ vault write secret/infra/network/allowed alreadyexists=maybe
Success! Data written to: secret/infra/network/allowed

I got blocked from creating a path inside of compute, and I didn’t need secret/infra/network created before making a child path. That infraadmin account is really not needed at all. Let’s go ahead and try infracompute.

$ vault auth d156326d-1ee6-7a93-d9d3-428e2211962d # auth as infracompute
$ vault write secret/infra/compute/obm/idrac/oem username=root password=calvin
Success! Data written to: secret/infra/compute/obm/idrac/oem
$ vault read secret/infra/compute/obm/idrac/oem
Error reading secret/infra/compute/obm/idrac/oem: Error making API request.

URL: GET http://vaultcore01.pool.lab.ytnoc.net:8200/v1/secret/infra/compute/obm/idrac/oem
Code: 403. Errors:

* permission denied

Oh my. I gave myself create, but not read permissions. New policies.

infranetwork.hcl

path "secret/infra/network/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "secret/infra/compute/obm/*" {
  capabilities = ["read", "list"]
}

path "auth/token/lookup-self" {
  capabilities = ["read"]
}

infracompute.hcl

path "secret/infra/compute/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "auth/token/lookup-self" {
  capabilities = ["read"]
}

Let’s update our policy list and cleanup.

vault auth b6eac78d-f278-4d32-6894-a8168d055340 # auth as root token
vault policy-delete infraadmin # delete unneeded infradmin policy
vault token-revoke d16dd3dc-cd9e-15e1-8e41-fef4168a429e # remove infraadmin token
vault policy-write infranetwork infranetwork.hcl
vault policy-write infracompute infracompute.hcl

Try again:

$ vault auth d156326d-1ee6-7a93-d9d3-428e2211962d # auth as infracompute
Successfully authenticated! You are now logged in.
token: d156326d-1ee6-7a93-d9d3-428e2211962d
token_duration: 2762315
token_policies: [default infracompute]
$ vault read secret/infra/compute/obm/idrac/oem
Key                 Value
---                 -----
refresh_interval    768h0m0s
password            calvin
username            root

And as network

$ vault auth 84faa448-20d9-b472-349f-1053c81ff4c9 #infranetwork
$ vault list secret/infra/compute
Error reading secret/infra/compute/: Error making API request.

URL: GET http://vaultcore01.pool.lab.ytnoc.net:8200/v1/secret/infra/compute?list=true
Code: 403. Errors:

* permission denied
$ vault list secret/infra/compute/obm
Keys
----
idrac/

$ vault list secret/infra/compute/obm/idrac
Keys
----
oem

$ vault read secret/infra/compute/obm/idrac/oem
Key                 Value
---                 -----
refresh_interval    768h0m0s
password            calvin
username            root

Camp night

My wife and son will be spending the night camping outside. I’d love to join them, but someone has to stay inside and have the entire bed to himself.

The Cheap Machete Problem

This weekend, I was finally clearing out the bit of walk between my pole barn and the hill behind it. Erosion has been filling it in, making it hard to get back there with any sort of push mower. Vines and brush have been filling in around it. I did what I could with the trimmer, then I went back in with the machete. Now, this machete was $6 at Harbor Freight and I haven’t used it much because the cheap plastic handle was starting to break.

I didn’t get a before photo, but one can see the kind of brush I was dealing with slightly up the hill. It wasn’t long before the handle was completely apart. The blade was fine, but the only thing holding the handle on the blade was.. my hand. I eventually ditched the handle, wrapped the base with 550 cord, and was able to finish, though my rope handle too started to unravel.

Last night I considered that today I would run up to Harbor Freight and pick up another cheap machete. They’re $6 and I don’t use them that often. But I would really like to be sure that the machete gets through a full job. There’s also this adage concerning tools: Buy the cheap one first and if you use it enough to wear it out, then buy a more expensive one. I went and looked on amazon, and they started off around $14. One these, I saw some reviews about the blades bending right away or rusting. Well.. this started me on a pretty dangerous journey to find the best machete to buy.

My first step on this journey was a pair of articles on bushcraftpro.com,
one focused on clearing brush and another on chopping wood. Both are written by the same author, and follow a similar format (buying a better machete for the husband to use). They certainly opened my eyes to the much larger world of machetes. The machetes that I’m used to seeing with a blade and saw on the back are common, but are never in the top of the list. A number of people prefer machetes over hatchets for chopping wood. There’s a youtube video showing propper chopping technique. I also learned that “the best” machete you can get is probably anything by Condor Tool & Knife. From their line, expect to pay around $50 to $90. This was a far cry from $14. I said “that’s too much, and this article is bunk” and started looking for other opinions. I was awoken to the idea that instead of the hatchet I keep in my car, I might want a machete instead.

I came across survival prep forums. People making their machete pick based on the impending collapse of society and/or the zombie apocalypse. I learned that machetes are “old hat” and I should consider Malaysian Parangs or Nepalese Kukris instead. I found the woodmans pal, which looks amazing and has stories off WW2 servicemen using them against enemies with katanas. I discovered Gerber used to make a good machete, but then they lowered the quality, so now it sucks.

After all this research, I went to bed thinking of what kind of awesome machete, Parang, or Kukri I would buy. Something I could keep in the car for brush clearing emergencies, perhaps with a sheath I could hook to my belt. Something I could use while camping or clearing out more brush from around the buildings. I had machete fever.

This morning, I woke with a slightly clearer head. I’ve used a machete about twice in the last 6 years. I’ve only once encountered a fallen tree on the road once that I can recall. In that case, I had no tools to clear it with, but was able to edge the car around it and leave the problem for the next unfortunate traveler. I keep thinking I’d like to go camping, but I haven’t done that in 20 years. I doubt buying a $80 knife would change that. If anyone learned how much I paid for my expensive knife, they would ridicule me and rightly so after a couple months, I would have forgotten all the details as to why this was so incredible and not be able to justify it other than ‘better handle’.

This has been about machetes, but it’s really a dangerous issue with learning too much. You start looking for something a little better. Then you learn that the one simple tool you never really thought about has an entire world of clashing opinions around it, and suddenly you start identifying with people who (claim to) use that tool for things that you never actually do.

So what blade am I planning to get? I’m not sure yet, but I do have it narrowed down.

  • The most sensible blade would probably be the $17 Whetstone Machete, recommended in the first brush clearing link above. It is full-tang with a reliable handle. The price is under $20, and it would probably do anything I plan to do it with.
  • The $20 Ontario Knife Co 1-18″ Military Machete has slightly better reviews than the Whetstone above, but some point out that the handle is a bit slippy.
  • The third (more expensive) choice is available for $40 and is Condor Eco Parang Machete. This is shorter, 11-inch blade and claims to have an unbreakable handle. “This tool’s high impact Polypropylene handle is strong and indestructible. These handles are molded directly into the machetes and knives blades making them impossible to separate”. It only has 11 reviews on Amazon, but after watching this youtube video and seeing them go from “meh” to impressed while chopping wood, shaving wood, and clearing brush. They mention the longer Condor Bushcraft, which is about $10 more.
  • The Condor Eco Golok as reviewed by the same guy can be gotten for $35 and also looks pretty impressive. I think I like the Eco Parang a bit more.

Frustrations

This morning I took a meter I was working on outside so I could take it apart and watch my son run around the yard. I planned ahead and took a box to hold the parts. After I had gotten a couple screws out, the wind picked up and blew my box into the yard. I can’t find the funny screws in the grass. I should have left them out of the box.

All The Changes

tl;dr

Here’s a quick summary of changes that have taken place:

  • A new site Nifty Noodle People has been launched
  • BCARS has been moved from mezzanine to wordpress and re-organized
  • A new community forum site has been launched: https://community.yourtech.us/
  • Comments for BCARS and YourTech use the community site now.
  • YourTech.us and YourTech Community now live on a dedicated VPS.
  • YourTech.us is now using WordPress, though all content is still generated in Markdown.
  • Soon, other sites will migrate as well.

Launching A New Site

Over the course of this last week (really a good bit of this year) I’ve been doing a lot more web work. In February, I launched Nifty Noodle People, an event website to promote BCARS‘s rebranded Skills Night 2.0. After trying many, many different systems, I settled on WordPress. WordPress is something I moved away from back in 2012. However, for a single purpose site, WordPress really impressed me. It impressed me so much, that I decided that I should redo BCARS site under wordpress as well. I had been using Mezzanine, Django-based CMS to manage their site and mine. But Mezzanine has been showing its age and often causing more problems than it’s worth when it comes to doing updates or adding things like an event calendar.

BCARS Changes

I setup a BCARS development wordpress site and started importing content into it. I spent a lot of time looking at different calendars. For Nifty Noodles, I had used The Events Calendar, and it’s a really nice calendaring system. But when I was trying to utilize it for BCARS, I ended up not liking the formatting options. I went back into research mode and ultimately settled on time.ly. I even picked up their Core+ package which lets me re-use vendors and organizers. This let me add in recurring events like meetings and weekly nets, and it allows people viewing the site to filter between regular and featured events (like a VE session).

As I was secretly working on this, it was brought up at a club meeting that the club would like to see a way to buy and sell gear on the site. So I added bbPress forum to the development site. Then I launched it silently on April 24th. It has gotten pretty solid reviews from people visiting it.

Server Move

As I was doing all this work, I observed that my Dreamhost VPS was prone to crashing. I also made an alarming discovery that I was paying a lot more each year than I had remembered. Also, I often get issues with it running out of memory and getting rebooted. I decided it was time to go searching. I had stuck with Dreamhost because of their nice control panel. They made it easy to spin up new sites, sub-domains, and “unlimited” everything. But it’s time to move on.

I looked at web hosts, then I looked at plain VPSes. I discovered that OVH had some really good pricing on SSD VPSes. A couple years ago, I would have bulked at “wasting time” managing a server in order to do something simple like pushing web content. But my skills with config management have come a long way over the last 5 years. I decided I would use Ansible to manage the VPS and use all the myriad of roles out there to do so. I’ll hopefully write more on that later. But in short, I’ve got roles installing mongodb, mysql, nginx, letsencrypt, and managing users. I couldn’t find a suitable role to manage nginx vhosts, especially in a way to start with a vhost on port 80, and not clobber the information letsencrypt puts in when it acquires a certificate. I hope to make a role that maintains http and https config separately, only putting in the https configuration if the certificate files exist.

But I digress.

Community Forums

During all this, I have been giving lots of thought to moving YourTech to wordpress as well. It’s a bit more challenging because I write all my notes in Markdown, which I then convert into posts. I started markdown blogging in 2012, and have shifted platforms several times since, most recently on Mezzanine. I was also thinking of better ways to engage the audiences of YourTech, BCARS, and Nifty Noodles. I had come across this article about replacing Disqus (which I had used) with Github comments. While I liked the idea, I knew it wouldn’t work for my goals. I kept coming back to forum software. I found three modern variations Discourse, NodeBB, and Flarum. Of the three, I like Flarum the best. Unfortunately, Flarum is still under heavy development and not recommended for production use. The authors can’t yet guarantee that you’ll be able to preserve data through upgrades. They want the flexibility to make changes to the database structure as they develop features. So I went to the next best, which is NodeBB.

NodeBB has a blog comments plugin that allows you to use a NodeBB to handle comments on your blog. The pieces all started coming together. I installed NodebB on my VPS as https://community.yourtech.us/. I changed the links on BCARS forums to point to this new community site, and integrated comments for BCARS.

YourTech Move

This weekend, I decided to pull the plug on YourTech.us and migrate it simultaneously into wordpress and into the new server. I new this would cause downtime, but since my blog is not commercian, and not exactly in the Alexa Top 500, I wasn’t too concerned. If anyone did notice downtime between the 5th and 7th, let me know below.

The move was not without hitches. I did have a markdown copy of all my posts, but I had to add yaml frontmatter to the top of them for github wordpress sync to work. Then I discovered that the plugin ignores my post_date and just makes all my posts match the time of the sync. Also, using the same repository I had been using in development caused issues as well. But eventually, I got all my posts imported with their original post date.

What I didn’t import was my resume and personal history. My contact page I did import, but it is rather out of date, so I feel I should update it soon. I want to rethink what I have on all three pages and how I present them, so that’s a future project.

Finally, I discarded the handful of disqus comments I had and integrated the comment system with YourTech Community.

Future Plans

  • I still need to migrate BCARS, Nifty Noodle People, and other sites away from Dreamhost. But I hope those moves will be pretty painless since it will be direct copy and DNS change.
  • I made YourTech.us look similar to how it did before the move, but I am not sure I’ll keep that look going forward.
  • Once Flarum becomes more production like and they build a NodeBB importer (and comment integration), I’ll quite possibly move to that.
  • Ultimately, I hope these changes will motivate me to write more frequently, now that I can easily post from my phone or web.

things are in a state of flux

UPDATE: Content has been re-added, but the published date information is still being corrected.

I am migrating the site to a new server and from mezzanine to wordpress.

There’s always a few thins to work out, and I should be able to restore the content sometime this weekend.

Cable Combs

I’ve been incrementally updating my home lab, and now I really wish I had a 3D printer. These cable combs look like they would be awesome for when it comes to getting the server rack re-organized.

http://www.thingiverse.com/thing:1320948/

One commenter said that the ones with a zip-tie slot will also fit nicely into a rack’s square hole. I do have a 3D printer on my “big ticket wish list” but I don’t think it’s in the cards for this year.

Down 6 pounds

At the beginning of this month, I said enough is enough and I forced myself back onto the slow carb diet. Slow carb is really another fancy name for low carbs, but it makes a distinction between complex and simple carbs. You also get a cheat day.

  • Rule 1: Avoid “white” carbohydrates
  • Rule 2: Eat the same few meals over and over again
  • Rule 3: Don’t drink calories
  • Rule 4: Take one day off per week

I have been almost exclusively eating steak salads (from Sheetz and Ed’s) and avoiding soda/sugar drinks (including diet). I had one cheat day so far, where I got to have ice cream and tons of pasta (I had it coincide with my Nifty Noodles and Drones Day). During that day, I was surprised that I didn’t gain any weight. The last time I was on this diet, I would loose a couple pounds during the week, then gain most of it back on cheat day, and average out the week with a loss of one pound.

I know for long term weight loss, it’s recommended to focus on 1lb a week, but I’ve been reading up on this and more and more people are agreeing that a quick drop in weight at the beginning is more motivating. After loosing 6.6lbs in 2 weeks, I have to agree.

weight loss chart

Offline Social Networking

A few days ago, I read a blog post by André Staltz about “AN OFF-GRID SOCIAL NETWORK“. I immediately intrigued and I’ll tell you why. Amateur Radio. The bulk of all amateur radio traffic takes place as either phone (voice) or CW (morse code). Textual communication is only a portion of the traffic. I have been on and done PSK31 to send real time messages to people that are listening right now. I have used WinLink to send email from my computer, over the radio, to a packet station in another state or another country. I am very active with APRS to send text messages over VHF (think real-time twitter for amateur radio). Actually, on thinking about about how APRS beacons go every 10 minutes, and beacon stations in general, I guess those do push more than voice does.