Tips, Tricks, and Notes on running RAID1 and RAID5 on XCP-NG: Part 1

Recently when installing XCP-NG on my home server, I ran into some persistent issues with setting up RAID1 on my boot disks, and setting up RAID5 for my secondary/bulk Local Storage Repository. Here are my notes and tips on succeeding with an installation such as this.

I have a chassis with 2x400GB SATA disks in a RAID1 array used for the boot disks that hold the OS and the first Local Storage Repository, and 3+ 2TB SAS disks in a RAID5 array with LVM on top, serving as both a Local Storage Repository as well as a bulk storage served via SMB/CIFS/NFS as a sort of NAS.

Installation Prep

Starting from scratch, we might want to wipe all the partition tables, boot records, and/or RAID superblocks on each target disk. This is optional, but if you know you don’t care about the data on your disks, it should help ensure success in case you had a prior installation of a RAID superblock or the GRUB bootloader on any of the disks. If you care about the data on any of the disks in your system, power off, unplug those disks, and start over. Boot up the installation disk, and type Alt+F2 to get to the console, then run the following:


for dev in `ls /dev/md*` ; do mdadmin --stop $dev ; done

for dev in `ls /dev/sd[a-z]+` ; do echo "Wiping Partitions and MBR on $dev" ; dd if=/dev/zero of=$dev bs=512 count=1 ; done

Next, reboot to start the installer over again, ensuring we start with a clean slate. Run the following to verify:


You should see no partitions or raid labels (ie, md127) on any of your target disks. If you do, you may need to re-run the above and try again.


Proceed as normal with the installation prompts, selecting the Software RAID option when it comes up.

Select the two (or more) disks you wish to add to the RAID1 array, then enter Create

Select your RAID disk as the install target (usually md127)

Select your RAID disk for your local storage (usually md127)

Proceed as normal with installation

When the install starts, type Alt+F2 to go to the CLI console

Run this to see the RAID resync process:

while true ; do tput clear ; date ; mdadmin --detail /dev/md127 | grep -v ^$ | grep -e ^ -e "Resync Status.*"; sleep 10 ; done

You should see a line starting with “Resync Status” indicating the percentage complete.

You may type Alt+F1 to go back to the installation progress screen.

Once the installation is complete, DO NOT REBOOT YET! I suspect this was the cause of an issue I had on one of my attempts doing this. The RAID1 array needs to finish syncing, or else you may be missing the required MBR/GRUB information on one of your disks, and your system may fail to boot if the non-synced disk happens to be the first in your BIOS/HBA boot order. Type Alt+F2 to go back to the CLI console and re-check the Resync Status progress. Once it reaches 100% complete and is in a State of “clean”, you may proceed.

Type Alt+F1 and finish the installation as normal.

Part 2 coming soon.

A new era, a new look

Well, the Atathualpa theme has served me well since I first migrated my blog to WordPress from Slashcode (yeah, crazy). However we live in a new world now, with a growing majority of web traffic coming from mobile devices. It’s time I got with the times and used a responsive design theme. So I’ve made a switch to the TwentyFifteen theme. I’m going minimalist for now, and will likely bring modifications back in as I see value.

Jetpack Broke My Comments


So apparently “JetPack Comments” broke my comments, causing recent comments to get posted to the wrong thread. They claim it’s not their fault, saying that it’s due to a lack of implementation of the comment_form() function in WordPress, but Atahualpa seems to support this just fine. Well, anyhow, for now I’ll be disabling JetPack Comments on my blog. So if you want to comment, you’ll have to sign up. I know. It’s a pain. I’ll get it fixed.

Update 6/10/2015: After switching themes to TwentyFifteen, the problem seems to be gone. JetPack Comments re-enabled.

Adfree Breaks Pinterest on Android

If you’re like me, you like keeping your Android device screen free from ads. AdFree from BigTinCan is an invaluable tool in assisting with this by customizing your hosts file on a rooted Android phone so that any ad network links get redirected to your phone, effectively disabling ads. The side effect of using host-based ad blocking is that sometimes valid sites get blocked as well.

“I don’t always do Pinterest, but when I do, I prefer pinning homebrew stuff.” And unfortunately, Adfree blocks pinning on Android. You’ll notice this when doing any pin outside of a re-pin (ie, within your pin feed). The app will churn saying it’s finding images, but then finally fail with the popup error “Sorry, couldn’t find any pinnable images on this page”. The issue is that the Pinterest app requires access to a few hostnames that Adfree hijacks:


This solution to this is fairly simple. Thankfully, BigTinCan offers an option to set up a customizable exception list but of course you’ll have to register for a free account. Once you have registered, add exceptions for each of the hosts above. Then sign in to your account on the AdFree android app and update your hosts. You should now be able to pin to your heart’s content.

Let me know if this helped you!

Pebble Smartwatch Skins

One of my biggest beefs with the Pebble is the plastic case. These guys have a solution to not only the scratchability of the case, but the plain black style. The woodgrain one looks really nice, and is probably the one I would go with if I were to get a Pebble.

AHK Script to Clean Up Dead PuTTY Sessions

I have a love-hate relationship with tabbed interfaces. On one hand, they keep my taskbar clean. On the other, they merely scuttle my window-hoarding behaviors into a single window, where the mess grows into a pile of outdated, unused tab sessions (i.e., Firefox). I also have need for some types of programs to be opened in separate windows on a regular basis, such as SSH sessions, where I often work on multiple related tasks which have to be monitored simultaneously. In those instances, the number of PuTTY sessions I have open (and have disconnected) can quickly grow out of control, clogging my taskbar.

PuTTY Fatal Error

Here’s a quick script I wrote to clean up any inactive/disconnected PuTTY windows: Cleanup PuTTY Windows

Fix for Chromecast “No Cast devices found”

Every so often I run into a Chromecast user who says they can’t get their PC to find their Chromecast. More often than not, it’s due to a bug in how the Chromecast plugin handles multiple network connections. The Chromecast plugin assumes that your primary active network connection is on the same LAN as your Chromecast dongle. Obviously this is not always the case, especially in the case of VPN users. What you have to do is move that connection to the top priority in your network connection list.

Here’s how you fix the dreaded “No Cast devices found” issue. This is for Windows 7, but probably works with 8 and Vista as well:
1) If you are connected to a VPN, disconnect it and try detecting the Chromecast again. Still not working? Proceed…
2) Go to the Network and Sharing Center
3) Click “Change Adapter Settings”
4) Click the Advanced menu and select “Advanced Settings…” (Alt+N, S)
5) Find the network connection that shares the same LAN as your Chromecast and select it
6) Repeatedly click the green up arrow until that connection is at the top of the list.
7) Click OK to close the dialog and apply the new connection priority settings
8) Try detecting your Chromecast again

If you use a VPN, you may be wondering why you have to disconnect. The answer is that many VPN clients manage the connection priority of the virtual adapter they create and automatically moves it to the top of the list. It’s best just to disconnect and try again instead of battling that behavior.

Good Luck! I hope this helps someone out there!

If you found this helpful, maybe you’d like to send a thank you from my wishlist?

Brewblog: All-Grain Bavarian Hefeweisen (Northern Brewer)

Just a quick update on my homebrewing adventures.

Finally got around to doing my first AG brew using my new mash tun, using the Northern Brewer Bavarian Hefeweisen kit. The crushed grain sat in my garage for a couple weeks, then got tossed into the kegerator for storage for the next 4-5 months. Hopefully the grains didn’t go stale and affect the flavor of the malt. As this was my first AG experience, I had a slight hurdle with the volume of the strike (?) water. I was attempting to do the more complex multi-step protein/saccharide rests, and may have started out with far too much water. Thus I ended up with about 9 gallons of liquid that needed to be boiled down to ~5. I still don’t fully understand what grain/water  target ratio I should be using at mash-in. Needless to say, I burned off a good amount of propane just boling off those 4 gallons of excess liquid.

I’m unsure of how to properly calculate mash tun efficiency, however what I do know is that the listen target for original gravity was 1.049, and I ended up with 1.046. I was hoping for higher, as I did rinse the grain pretty well (or so I thought), and by comparison, my experience with the extract version of this kit has resulted in an OG of as high as 1.052. The comparison might not be valid though.

Fermentation was very active, as usual, finishing up it’s frothy activity within the first 36-48 hours. This time around I didn’t have any issues with blowoff.

So, with any luck, I should have a perfectly drinkable beverage in just under 4 weeks. Now to figure out where to re-fill my 10# CO2 cylinder.

Here’s my current brewhouse/pub status:

On deck:        Pumpkin Ale (Indie)
Primary:        Bavarian Hefeweisen (Northern Brewer)
Secondary 1:    Empty
Secondary 2:    Empty
Keg 1:          Tapped-out
Keg 2:          Tapped-out
Keg 3:          Tapped-out
Keg 4:          Tapped-out

HowTo Resolve StartSSL (StartCom) Domain Blacklisted: Domain appears on a blacklist

Does this look familiar to you?


Welcome to my world. Not sure at this point how I got on this list, how to get off it, or even where this list is. But perhaps my findings will help you resolve the same issue for your domain. At this point, my suspicion is that it’s due to and odd report from Google Safe Browsing that “Yes, this site has hosted malicious software over the past 90 days. It infected 0 domain(s), including .”. It would be great if I knew what the malware/badware is/was so that I could remove it. Even more odd is that my supposed infection infected no other sites.

Oh well. More to come…

Update: I’ve emailed “Certmaster” and they responded letting me know that they see my domain on Google’s Safe Browsing blacklist results. Oddly enough, here are the results:

I see the report that “Yes, this site has hosted malicious software over the past 90 days. It infected 0 domain(s), including .” What’s odd about this is that when I check my Google Webmaster tools, the site reports that “Google has not detected any malware on this site.”, and it seems I’m not the only one. Not sure if I’m just bitten by a previously unseen issue that I’ve since cleaned up with WordPress updates or what.

Given the date that is shown above (2013-05-04), I’d guess that the “past 90 days” implies I’ll have to wait until 2013-08-04 for this status to clear. I guess that’s the penalty I pay for lack of diligence in monitoring the updates and health of my server up until then. If you’re saying to yourself “I can’t wait that long!”, you do have the option of paying StartSSL the fee required for them to manually intervene in what would otherwise be an automated process. I choose to wait it out: I don’t really need SSL for anything practical. For my purposes, it’s just for the sake of writing articles like this: research and writing howto’s based upon my experiences. So I’ll be waiting out the presumably prerequisite “90 days” for the sake of research.

See you on 8/4 with an update!

Update 8/15: As the saying goes: “Time heals all wounds”. I’m now off the naughty list for Google. Now to (re-)try obtaining a cert from StartSSL…

Accidental Evangelist

It’s been a while since updating my blog with anything overtly Catholic. I guess work, family, and life in general started to consume far more of my time in last several years. Non-geek-related posts have been sparse to non-existent. It’s been bothering me some privately in recent months with elections coming, and now gone. There’s so much topical matter to cover, and unfortunately, most of it gets posted to Facebook where my audience is limited (and intentionally so).

If been feeling that, as a result, I haven’t been doing a very good job of being an active evangelist for Catholicism. It’s been taking a back seat to being charitable with my technical knowledge. I’ve gained a lot of knowledge from peers in the past 15+ years, and this blog is one of the ways I like to “pay forward” that generosity.

Which brings me to this blog article by Creative Minority Report: Accidental Evangelism.

I think my sense of charity directly, and most specifically a personal impressibility to be actively charitable stems from my Catholic upbringing. I’d could only hope that some person reading this blog would be converted (at least in part) by my actions.

May God bless, and the grace of God be with you!

Solution to the NVidia Gray/Grayscale Screen Problem

Today I came in to work to find that the video output for my Dell Latitude e6520 laptop’s NVidia head was displaying in black and white. At first I thought that the problem was a driver bug, something wrong with the video memory, or a faulty display. But eventually I found out that, somehow (without any user interaction with the applicable setting), the “Digital Vibrance” setting was set to 0%, when it should be 50%. Below you’ll see a simple annotated screenshot showing where you can quickly fix this.

Nvidia Gray Screen Problem - Annotated

Good luck!

Streaming RTMP with VLC and RTMPDump

This quick post is as much for your benefit as for the benefit of my memory…

To stream RTMP with VLC, you’ll need rtmpdump, which you can get here: I used, though you may be able to use the latest version. I also had VLC 2.0.6 32-bit installed. Once installed, you can run the following from a cmd window:

rtmpdump.exe -r “rtmp://” -v -o – | vlc.exe –

This worked nicely for me. YMMV. Good luck!

If you found this helpful, maybe you’d like to send a thank you from my wishlist?

FFMPEG “Server error: Not Found” with Short URLs

Just a quick post about a problem I helped a buddy of mine resolve. He was setting up a Helix media streaming server, and was trying to capture the stream data to a file with the following command:

ffmpeg -i “rtmp://” out.flv

The result was this error in the output:

[rtmp @ 0x28e1dc0] Server error: Not Found

Oddly enough, the connection information shown on the Helix console showed that a strange URL was being requested. Upon further investigation with Wireshark, I found that this was the request being made.


Note that “\360” is a hex character. For some odd reason, it would appear that ffmpeg improperly handles short URLs, inserting a string of “\360xw0”. If you pad the URL with the current directory “./”, then the request succeeds:

ffmpeg -i “rtmp://” out.flv

This results in a request of


Which worked fine in our environment.

For future reference, I was running this ffmpeg version (on CentOS 6.4 x86_64):

ffmpeg version N-53616-g7a2edcf Copyright (c) 2000-2013 the FFmpeg developers
built on May 29 2013 00:19:54 with gcc 4.4.7 (GCC) 20120313 (Red Hat 4.4.7-3)

So if you’re running into the “Server error: Not Found” error on a known good URL, try padding the path of the stream with “./” and see if that fixes it for you. I’m guessing this is an ffmpeg bug, but don’t really have the access to a streaming server to troubleshoot and submit a bug report. From the time that I did have, it appears that it’s related to the .flv extension in the rtmp URL. If you drop the extension, the URL can be of any size.

Brewblog: Rubbermaid 50 Quart Mash Tun

After several months of lingering in my garage, I finally finished my 50 qt. Rubbermaid mash tun. The cooler I used is the one shown below:



I picked it up at my local Menards for about $30. You should be also able to find them at Walmart for about $40. I chose this cooler because I’ve purchased a few of them over the past years, so I know they’ll likely be making them for a while. Also, the design is relatively easy to build a mash tun manifold for. And, according to this chart, I should be able to make up to 10 gallons of wort comfortably in there, allowing me to eventually make double batches.

The manifold is a design that I came up with, and eventually I’ll get around to actually measuring and posting dimensions of the piping so you can build one too. I used under 8′ of copper pipe for it, and should have enough length left over to eventually build a fly sparge manifold (or at least a start on it). The design relies on two lengths of  bare copper wire to hold the whole thing together, and two small segments of tubing at the back of the cooler keeps the manifold  securely wedged in position. I’m not averse to soldering, or even all that concerned about contamination from the silver-based solder. My main reason for going solderless is to make it easy to clean. I can literally disassemble the entire manifold into individual components, ready for a thorough brushing if desired.


Recovering the PPP Username and Password from a Centurylink Actiontec C1000A

Some time ago I wrote up a similar procedure to recover a password from an Actiontec M1000 back when Centurylink was known as Qwest (gotta love rebranding). Back then, Actiontec left the operating system a bit more open, actually placing the PPP credentials in a flat file in /var/tmp/. Nowdays, Actiontec tries to obfuscate/encrypt the password in a config XML, making it just difficult enough for most people to give up on the idea of recovering the password.

In addition to this, it would appear to the casual telnet console user that the commonplace busybox shell had been removed or made inaccessible, removing the ability to peer into the embedded linux operating system underneath, and replacing it with a stripped-down properietary shell with limited commands. However, dig a little deeper by trying the undocumented “sh” command, and you’ll find that busybox is alive and well on this device, exposing the configs and services that support the router’s functions. Since the configs contain the encrypted password, we can’t directly extract them from there, however fortunately for us, pppd (the service that authenticates and creates the DSL connection) requires the password to either be kept in plain text in a flat config file (not the case here) OR have the password specified on the command line. The “ps” command on these devices has been handicapped to only display 80 columns, leading to output similar to this:

1623 admin      1144 S   pppd -c ppp0.1 -D 0 -i ptm0.0 -u “your___username@qwe

At first, when I saw this, I figured it was another dead end, until I realized that “/proc/(pid)/cmdline” displays the command line of any running process. And fortunately for us, Actiontec left “pidstat” enabled in busybox, making reading those command line arguments fairly simple with a single command:

 > sh -c “/usr/bin/pidstat -l -C pppd”
Linux 2.6.30 ((none))   05/23/13        _mips_  (2 CPU)

14:59:38          PID    %usr %system  %guest    %CPU   CPU  Command
14:59:38         1623    0.00    0.01    0.00    0.01     1  pppd -c ppp0.1 -D 0 -i ptm0.0 -u “” -p “AbCDEfgH” -f 0 -k -P “AbCDEfgH” -M 1492

And there you have it! In this case, I’ve altered the password output (shown as AbCDEfgH) to protect my own privacy, but it’ll look similarly like jumbled letters and numbers on your command line. Also note the “” username. Qwest used to be the LEC in my area, and likely the transition hasn’t been made to the newer branding in the back-end systems. Your situation may be different.

Also, in case you hadn’t figured it out by now, you’re going to need telnet console access. My modem’s console admin password was not the default “admin”, nor the admin password listed on the box. I had to log into the web UI, enable remote console, and (re)set the console password. After that, you should be able to log into the telnet console with “admin” and the password you’ve set.

Lastly, one might ask “Why would I want to obtain this password?”. My personal reason is that eventually I want to replace this leased modem with one I’ve purchased, and use it in transparent bridging mode (using RP-PPPOE to terminate the PPPoE connection and it’s leased static IP on my firewall). When that day comes, I’d like to be able to do the swap without interacting with Centurylink. Call them by whatever new brand they’ve been changed to, Ma’ Bell is always a pain to talk to.

Good luck to you!

If you found this helpful, maybe you’d like to send a thank you from my wishlist?

Pitfall-Free Howto/Guide to StartCom/StartSSL Class 2 Organization Validation/Certification

…in just three not-so-simple steps.

A couple quick notes before we begin: I threw this article together over a period of weeks, so the layout is a bit… odd. At some point I’ll come back to this article and clean it up, however for now, I think it does the job of conveying the process to a new StartSSL user. Also, I use the names “StartCom” and “StartSSL” interchangeably, so don’t look any deeper into the name usage than that.

A few weeks ago, I started the process of renewing several StartCom certs for my employer, and started to become familiar with the processes and pitfalls of identity and organization validation with StartCom. After delving into the process head-first, it became evident that this might not be as straight-forward as one might expect. However, I think that once you know what to expect, the process should go much more smoothly.

Also, please note that this how-to merely documents what I did to get a cert, and what pitfalls I ran into along the way. Therefore, your requirements and path may (and probably will) vary from mine. For instance, the tax document I submitted was a “State of Delaware Annual Franchise Tax Report”. This document lists all of the items shown on StartCom’s requirements. Your document(s) may not.

First, let’s understand the steps required in obtaining Class 2 Org validation. Basically, you keep escalating your level of validation, providing increasingly trusted levels of personal and organization documentation as you go. Here’s a rough outline:

  • Get Email Address validation with StartSSL (free)
    • Sign up
    • Validate email
    • Get personal client cert (for browser) via website
      • Back it up!!!
  • Get Personal Class 2 validation (~$60)
    • Submit 2 forms of ID
      • 1 Photo ID (i.e.: Driver’s license)
      • 1 other form of ID. May be photo (i.e.: Passport), OR non-photo (i.e.: birth certificate)
    • AND submit either:
      • Phone bill showing your name, current address, AND (most importantly) phone number.
        • May be land line or cell


      • Request for certified mail validation
        • Delivery will take ~2.5 to 5 weeks (from Israel)
          • If expedited service is required, you may additionally request express mail (~4-5 bus. days) for approx $30.
    • Wait for call or letter, and verify provided code on StartSSL website control panel.
  • Get Organization Class 2 validation (~$60)
    • Submit tax document which contains:
      • Name of CEO
      • Names of Directors
      • Co. Address (not sure if this is an actual requirement)
      • Co. Phone number (not sure if this is an actual requirement)
    • AND submit StartSSL’s “Delegate Authorization Letter” ( with:
      • Your name, title as delegate
      • Name and signature of CEO, President, or Director (CFO also appears to work. Other C?? titles may suffice)
    • Have your signatory receive a phone call from StartCom to verify your authorization.
  • Validate your domain(s)
    • Perform the “Domain Name Validation” validation wizard for each new or expiring domain
    • Confirm validation using code sent to email
  • Generate Certs
    • Perform the “Web Server SSL / TLS Certificate” certificate wizard
      • Only new/expired certs or certs expiring within 2 weeks may be renewed (when the existing cert is with StartCom)
    • Generate password-protected key / cert pairs
      • Keep that password safe and backed up!
    • Save key to .key file, cert to .crt file (ie,,
  • Install Certs
    • Decrypt (strip password protection from) key file
    • Place decrypted key and crt files on webserver
    • Configure Apache SSL
    • Reload Apache
    • Verify correct cert in web browser
  • ???
  • PROFIT!!!

And now for the more elaborate explanation:

1) Get your email address set up, and obtain a website authentication certificate. This is free. Go to the Control Panel and click Sign Up. Fill out the forms and submit. Soon you should receive an authentication code via email. You will submit this to the StartCom site, and then wait for another email that has a verification link, which you’ll click. Once that’s done, you’ll get a personal client certificate for your browser. Follow the instructions in the email/website on how to backup this certificate. Don’t skip this! Think of this as cert the key to your account. If you lose it, you’re likely going to have to a hard time regaining access. Back it up to an external storage medium (perhaps even encrypted as well – TrueCrypt is good for this).

2) Get yourself a Class 2 validation. This will cost you ~$60 (with one potential caveat, described below). This is where things start to get more complex. You’ll need two scanned or photographed forms of documentation that prove that you exist, AND a means of verifying that the person submitting the documentation is actually you. For the two forms of identification, one must be a photo ID (afaik), and another can be another photo ID (such as a passport) OR something that simply proves you exist, such as a birth certificate (this is what I used). Submitted photos/scans must be of “high” quality, but less than 1400×1400 resolution. The second half of this, and this is where I got hung up, is the verification that the person submitting the information is YOU. StartCom apparently trusts phone companies, because you “just” need to submit an invoice/bill showing your name, address, and phone number. This can be either a scan/photo or a PDF. This is where I got hung up. My current cell service is not in my name, and I have Ooma VOIP for home phone service, which doesn’t do traditional paper or PDF invoicing. Attempts to submit website screenshots, or PDFs of a webpage will likely be rejected. StartCom needs to be able to verify that you own the number you say you do, and then call you at that number to verify that you submitted the request (presumably with a code conveyed by voice, then submitted to the StartCom website). In lieu of a verified phone number, your next/only option is a registered mail letter from StartCom (in Israel). They claim this takes 3-5 business days. That may be true if you lived in Israel. I live in the US. Internet rumor has it that this letter takes ~5 weeks to get to the US destinations. I’ll see if one of my letters end up taking this long (see update below… took ~2 weeks). The alternative is to request express mail delivery, which costs ~$30. This will take approximately 4-5 business days. I opted for this to expedite the process. Once you receive the letter, it will contain a code which you submit via the StartCom website control panel. Shortly after, StartCom should send you confirmation that you have been personally Class 2 verified.

3) Get your Organization Class 2 verified (another ~$60). This part is still in progress for me. I will be updating as I go along. First step appears (at least for now) that I have to obtain a yet-to-be-identified tax document from our finance/accounting department that proves that our company is recognized to exist by either the State or the US. Also, it’s not clear if I, as an employee of the company, can submit the tax documentation and as a result get access to create these certs, OR do I also need to submit an authorization form that delegates me to make cert changes on behalf of the company. Sure would be nice if StartCom would clear some of this up on their website, or at least on their forums.

More on this as the mystery unfolds…

Update 5/28: A few updates

  • Today I received my initially-requested snail-mail letter from StartCom. The letter was requested late on the 10th, so that means it takes 14-16 postal days (Mon-Sat) to get to Minneapolis, MN. But of course, YMMV.
  • In the case of my company, the tax document used was a Delaware Annual Franchise Tax Report (as the company is incorporated in Delaware). Your document will almost surely be different. My recommendation is to ask your accountant or executive for a copy of the Articles of Incorporation applicable to your state.
  • Unless you are the CEO, President, or a Director of the company/org you are applying for, you’ll need to fill out the  StartSSL Delegate Authorization Letter, printed on company letterhead, and have one of the aforementioned individuals sign it. Currently I’m attempting to use our CFO’s signature to see if that is sufficient. If not, I’ll be going to CEO to get the required signature.

Update 6/4: Eventually I got a signature from our CFO. I submitted the document to StartCom, and within a few hours I got an email back they they were unable to reach anyone at the listed office number. Presumably they tried to contact our CFO. I asked them to try again, and soon after got an email stating that the Class 2 Org Validation is complete! Once that was done, I was able to log into the StartCom control panel and perform Domain Name Validation(s) for each of our domains. It appears they offer a number of hostmaster/postmaster addresses as options to send a validation email to your domain. In addition to our other TLD’s, they offered Not sure how they linked them to our main corporate domain (possibly public WHOIS data), but they did. Up next, cert generation!…

Update 6/7: This ended up being the most straightforward process of the entire adventure. In my case, the web service in use was Apache. This process will likely vary for an MS-based/IIS server. Once your domain(s) is/are validated, you can go back into the StartSSL control panel, go the the certificates wizard, and select “Web Server SSL/TLS Certificate”. Set a password for you keyfile and generate the key. Keep good tabs on this password as it’s what decrypts your private key for use on your webserver. Save the resulting key text out to a file named something like “”. Continue on and select your domain, and enter/add your subdomain(s) or subdomain wildcards as needed (see notes below regarding restrictions). Once the cert is generated, save the text into a file named something like “”. Keep these files safe and backed up!

A few other notes related to this: Note that a subdomain wildcard only applies to the level of that wildcard. So for instance, * would cover, byt not For that, you need to specify wildcards for each recursive level, ie * Also note that domains that still have active StartSSL certs (or are not expiring within 2 weeks) are not able to be renewed. If you must renew them, you must revoke the existing cert, which costs approx $25. Be aware that you could still generate certs for individual specific (sub)domains instead to avoid going through the revocation process.

You can now proceed to install the cert on your webserver, however note that hte .key file is password protected/encrypted. You must strip the password in order to allow your web service to start up unattended. Here’s a quick command to do that:

openssl rsa -in >

You can then take the decrypted key and use it in your Apache config. Please note however that you must take care to restrict access to your server so that your private key is not copied by unauthorized persons. They alternative to this is to leave password protection on, but intervene manually by entering in the password each time Apache is (re)started.
So there you have it! At some point I’ll probably come back to this article and clean it up. For now, I just wanted to get the information out there to help my fellow interweb users. Good luck!

Routed OpenVPN HOWTO

This is my OpenVPN HOWTO. There are many like it, but this one is mine.

It seems every few months I get asked the question by one of my friends “How do I set up a VPN?”. Sometimes the person is looking to set up a MS VPN variant, other times, OpenVPN. The principles and concepts seem simple to me, now, however for someone new to VPN architectures and perhaps even routing, it can be confusing. This is my attempt to make the mysterious understandable. Since roadwarrior (individual laptop clients) configs are fairly well documented by the official OpenVPN wiki, I’ll concentrate on a simple routed, LAN-to-LAN VPN networking concept, and cover roadwarrior config as an afterthought.

My weapon of choice distro-wise is CentOS, however these instructions could be applied to any other distro (ie, Ubuntu) with a basic understanding of your particular platform’s networking configuration methods. Really, OpenVPN can run on just about anything, including Windows, if you feel so inclined. However, you’ll probably get the most bang for your buck (free) using Linux. In my test environment, running stock CentOS 6.4, I had the scenario below running on a server with less than 90MB of total system memory usage.

Why CentOS?
I have two main reasons for promoting the use of CentOS:
1) CentOS is based upon RedHat Enterprise Linux (RHEL), which currently enjoys status as the most vendor-supported OS in enterprise environments. Translation? Knowledge of an RHEL derivitaves like CentOS is marketable skill to put on your resume. Yes, Ubuntu is gaining popularity in tech circles, but still doesn’t compare to RHEL for vendor support. There’s a reason both VMWare and Citrix use RHEL derivitaves as their baremetal OS.
2) CentOS has long-term-support (LTS). I’ve used Fedora for years, and I enjoy(ed) playing with some of the bleeding-edge features it offers. But the bleeding edge is on a double-edged sword. Fedora has a relatively aggressive release and support schedule. Install Fedora X, and expect that Fedora Z will replace it in about a year, leaving version X effectively without support. This gets to be a pain when you need to “yum update” your system just a year or so after you installed it. CentOS however has a support schedule that will ensure you likely have updates for far more years than the lifecycle of your hardware. For example CentOS 6, released in 2011, reaches EOL in late 2020. That’s almost 10 years of support, on a free platform!

First, let’s diagram the network we are going to design:

Remote Office 1 (10.101.0.x/24)
VPN Tunnel (
Main Office (10.100.0.x/24)
VPN Tunnel (
Remote Office 2 (10.102.0.x/24)

For this guide, we’re going to assume you want all remote offices to have routing enabled to each other (via the Main office).

Note that the IP addresses above are just for example. You could create your own IP addressing scheme with FAR better utilization of the the private address space. The subnets I have used (ie 10.200.x.x/16) are just for increased clarity.

Each OpenVPN server/endpoint can have one interface (assuming you are doing NAT/firewalling elsewhere on your network), OR you could have dual interfaces: One on the LAN, the other on the WAN/Internet connection. It’s up to you depending on where you want to do your firewalling. For the purposes of this guide, we’re putting the VPN server behind another firewall.

On your firewall
Forward UDP port 1194 for your external internet connection to the internal IP of your VPN servers/endpoints, ie:

Main Office
Remote Office 1
Remote Office 2

If for some reason UDP/1194 were blocked by your ISP, you could switch to something like TCP/80, but for the purposes of this guide, we’ll stick with the default UDP/1194.

On each server/endpoint
Install your base OS. I chose to just install CentOS 6.4 minimal installation with the default options. Once installed, get networking set up and run

yum update

and get everything up-to-date.

Install the EPEL Repo configs

yum install

Turn off the built-in firewall (remember, we’re using a 3rd party device for this). You could leave it on, but then you’d have to configure it to pass traffic to/from your VPN tunnel interface. I’ll leave that up to you to figure out should you choose to do so.

chkconfig iptables off ; service iptables stop

Install the OpenVPN packages and dependencies:

yum install openvpn

Server (Main Office)

Copy the the easy-rsa environment to /etc/openvpn/easy-rsa, do some config prep, and alter the vars file

cp -rp /usr/share/openvpn/easy-rsa/2.0 /etc/openvpn/easy-rsa
cd /etc/openvpn/easy-rsa/
cp -rp vars vars.orig
cp -rp openssl-1.0.0.cnf openssl.cnf
vi vars

Go down to “KEY_COUNTRY” and edit the Country, Province, etc, down to the OU, ie:

export KEY_CITY="Minneapolis"
export KEY_ORG="Muchtall"
export KEY_EMAIL=""
export KEY_NAME=MuchtallOpenVPNServer
export KEY_OU=Muchtall

Now generate your server’s certificate authority:

. ./vars

Accept the defaults for the prompts (we already set them)

Now build the server key


Similar to above, accept the default prompts. You will have to answer “y” to the questions “Sign the certificate? [y/n]:” and “1 out of 1 certificate requests certified, commit? [y/n]”

Now generate certs/keys for each remote site

./build-key remote-office-1
./build-key remote-office-2

Rinse, repeat on the prompts.

And generate the Diffie-Hellman parameters:


Great! Our certificates are all set up!

Next time you want to generate a new client key, just run

cd /etc/openvpn/easy-rsa/
. ./vars
./build-key remote-office-3

Now, let’s set up the configs. There’s a sample config at /usr/share/doc/openvpn-2.2.2/sample-config-files/server.conf, however we’re going to set one up using this template, just to keep things simple:

port 1194
proto udp
dev tun
ca easy-rsa/keys/ca.crt
cert easy-rsa/keys/
key easy-rsa/keys/
dh easy-rsa/keys/dh1024.pem
ifconfig-pool-persist ipp.txt
# Tell clients that we can handle routes for these networks
push "route"
push "route"
push "route"
client-config-dir ccd
# Tell OpenVPN that it routes for anything within these subnets
keepalive 10 120
status openvpn-status.log
verb 3

Now create the ccd directory

mkdir /etc/openvpn/ccd

And lets create the client-specific configs to route each individual subnet to the respective site:

# In /etc/openvpn/ccd/remote-office-1


# In /etc/openvpn/ccd/remote-office-2

Good? Good!

Normally, IP subnets for the tunnels are allocated as new tunnels connect to the server. Let’s pre-set the IPs for each tunnel. This part isn’t necessary, however I like to do this to assist with clarity in troubleshooting with traceroutes. In /etc/openvpn/ipp.txt:


And re-set the SELinux permissions on the ipp.txt file

restorecon -v './ipp.txt'

Now we’re ready to start the the OpenVPN service up:

service openvpn restart

Check the syslog to see if anything serious got spit out:

tail -100 /var/log/messages

You should see something similar to this:

May 7 14:54:53 mainoffice openvpn[13362]: OpenVPN 2.2.2 x86_64-redhat-linux-gnu [SSL] [LZO2] [EPOLL] [PKCS11] [eurephia] built on Aug 10 2012
May 7 14:54:53 mainoffice openvpn[13362]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
May 7 14:54:53 mainoffice openvpn[13362]: Diffie-Hellman initialized with 1024 bit key
May 7 14:54:53 mainoffice openvpn[13362]: TLS-Auth MTU parms [ L:1542 D:138 EF:38 EB:0 ET:0 EL:0 ]
May 7 14:54:53 mainoffice openvpn[13362]: Socket Buffers: R=[229376->131072] S=[229376->131072]
May 7 14:54:53 mainoffice openvpn[13362]: ROUTE default_gateway=
May 7 14:54:53 mainoffice openvpn[13362]: TUN/TAP device tun0 opened
May 7 14:54:53 mainoffice openvpn[13362]: TUN/TAP TX queue length set to 100
May 7 14:54:53 mainoffice openvpn[13362]: /sbin/ip link set dev tun0 up mtu 1500
May 7 14:54:53 mainoffice openvpn[13362]: /sbin/ip addr add dev tun0 local peer
May 7 14:54:53 mainoffice openvpn[13362]: /sbin/ip route add via
May 7 14:54:53 mainoffice openvpn[13362]: /sbin/ip route add via
May 7 14:54:53 mainoffice openvpn[13362]: /sbin/ip route add via
May 7 14:54:53 mainoffice openvpn[13362]: Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
May 7 14:54:53 mainoffice openvpn[13369]: UDPv4 link local (bound): [undef]:1194
May 7 14:54:53 mainoffice openvpn[13369]: UDPv4 link remote: [undef]
May 7 14:54:53 mainoffice openvpn[13369]: MULTI: multi_init called, r=256 v=256
May 7 14:54:53 mainoffice openvpn[13369]: IFCONFIG POOL: base= size=16382
May 7 14:54:53 mainoffice openvpn[13369]: IFCONFIG POOL LIST
May 7 14:54:53 mainoffice openvpn[13369]: remote-office-1,
May 7 14:54:53 mainoffice openvpn[13369]: remote-office-2,
May 7 14:54:53 mainoffice openvpn[13369]: Initialization Sequence Completed
May 7 14:54:53 mainoffice kernel: tun0: Disabled Privacy Extensions

Once you’ve verified based upon the above output that everything is running fine, go ahead and mark the service to start automatically

chkconfig openvpn on

Client Configs (Remote Offices)
On each server, create the file “/etc/openvpn/”, and populate with the following:

dev tun
proto udp
remote 1194
resolv-retry infinite
ca ca.crt
cert remote-office-1.crt
key remote-office-1.key
verb 3

Be sure to change the cert name as appropriate.

Copy the ca.crt, remote-office-1.key, and remote-office-1.crt to the /etc/openvpn/ directory of the client. Repeat for Office 2.

Set the permissions on the key file so that it can’t be copied by non-root users.

chmod 600 /etc/openvpn/remote-office-1.key

Start the OpenVPN service

service openvpn start

Check the output of syslog for similar output:

May 7 16:26:39 remote-office-1 openvpn[1566]: OpenVPN 2.2.2 x86_64-redhat-linux-gnu [SSL] [LZO2] [EPOLL] [PKCS11] [eurephia] built on Aug 10 2012
May 7 16:26:39 remote-office-1 openvpn[1566]: WARNING: No server certificate verification method has been enabled. See for more info.
May 7 16:26:39 remote-office-1 openvpn[1566]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
May 7 16:26:39 remote-office-1 openvpn[1566]: LZO compression initialized
May 7 16:26:39 remote-office-1 openvpn[1566]: Control Channel MTU parms [ L:1542 D:138 EF:38 EB:0 ET:0 EL:0 ]
May 7 16:26:39 remote-office-1 openvpn[1566]: Socket Buffers: R=[229376->131072] S=[229376->131072]
May 7 16:26:39 remote-office-1 openvpn[1566]: Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
May 7 16:26:39 remote-office-1 openvpn[1566]: Local Options hash (VER=V4): '41690919'
May 7 16:26:39 remote-office-1 openvpn[1566]: Expected Remote Options hash (VER=V4): '530fdded'
May 7 16:26:39 remote-office-1 openvpn[1567]: UDPv4 link local: [undef]
May 7 16:26:39 remote-office-1 openvpn[1567]: UDPv4 link remote: x.x.x.x:1194
May 7 16:26:39 remote-office-1 openvpn[1567]: TLS: Initial packet from x.x.x.x:1194, sid=f439995e ac9dd302
May 7 16:26:39 remote-office-1 openvpn[1567]: VERIFY OK: depth=1, /C=US/ST=MN/L=Minneapolis/O=Muchtall/OU=Muchtall/
May 7 16:26:39 remote-office-1 openvpn[1567]: VERIFY OK: depth=0, /C=US/ST=MN/L=Minneapolis/O=Muchtall/OU=Muchtall/
May 7 16:26:39 remote-office-1 openvpn[1567]: Data Channel Encrypt: Cipher 'BF-CBC' initialized with 128 bit key
May 7 16:26:39 remote-office-1 openvpn[1567]: Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
May 7 16:26:39 remote-office-1 openvpn[1567]: Data Channel Decrypt: Cipher 'BF-CBC' initialized with 128 bit key
May 7 16:26:39 remote-office-1 openvpn[1567]: Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
May 7 16:26:39 remote-office-1 openvpn[1567]: Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 1024 bit RSA
May 7 16:26:39 remote-office-1 openvpn[1567]: [] Peer Connection Initiated with
May 7 16:26:41 remote-office-1 openvpn[1567]: SENT CONTROL []: 'PUSH_REQUEST' (status=1)
May 7 16:26:41 remote-office-1 openvpn[1567]: PUSH: Received control message: 'PUSH_REPLY,route,route,route,route,topology net30,ping 10,ping-restart 120,ifconfig'
May 7 16:26:41 remote-office-1 openvpn[1567]: OPTIONS IMPORT: timers and/or timeouts modified
May 7 16:26:41 remote-office-1 openvpn[1567]: OPTIONS IMPORT: --ifconfig/up options modified
May 7 16:26:41 remote-office-1 openvpn[1567]: OPTIONS IMPORT: route options modified
May 7 16:26:41 remote-office-1 openvpn[1567]: ROUTE default_gateway=
May 7 16:26:41 remote-office-1 openvpn[1567]: TUN/TAP device tun0 opened
May 7 16:26:41 remote-office-1 openvpn[1567]: TUN/TAP TX queue length set to 100
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip link set dev tun0 up mtu 1500
May 7 16:26:41 remote-office-1 kernel: tun0: Disabled Privacy Extensions
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip addr add dev tun0 local peer
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip route add via
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip route add via
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip route add via
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip route add via
May 7 16:26:41 remote-office-1 openvpn[1567]: Initialization Sequence Completed

And verify that the routing is taking place


Set the openvpn service to start automatically

chkconfig openvpn on

Repeat these steps for the Office 2 client. And verify that you can ping across both tunnels to Office 1

Roadwarrior Config (this is optional)
As an afterthought, I said I’d cover Roadwarrior configuration. Here’s a basic rundown:

– Generate a new cert/key pair for your username using the above ./build-key commands
– Install OpenVPN for Windows (if you’re on Mac or Linux, you likely know how to do this already)
– Copy the ca.crt, username.crt, and username.key files to “C:\Program Files\OpenVPN\config\”
– Create a config file named C:\Program Files\OpenVPN\config\ with these contents:

dev tun
proto udp
remote 1194
resolv-retry infinite
ca ca.crt
cert myusername.crt
key myusername.key
verb 3

Right-click on the OpenVPN GUI in the taskbar and click “Connect”.

Muchtall Window Arranger (Save Window Positions and Size)

Over the years, I’ve noticed that there’s one “drawback” to multiple heads/screens on a laptop workstation. I get in the habit of arranging my workspace how I like it, where eventually I have 8-10 windows in positions I would prefer to be permanent whenever I hook up my external 2 heads. When I undock, of course everything collapses back to my main display. When I re-dock, the windows remain in a cluttered pile on my main screen. In addition to this, changes in resolution can result in windows needing to be re-sized as well.

I’ve found one or two utilities over the years that have allowed me to save window positions and sizes, but nothing that handled it for me as automatically as I would have liked. So I wrote my own in AutoHotKey: the Muchtall Window Arranger.

It’s fairly simple, yet powerful if you want it to be. Since it’s written in AutoHotKey, you can write custom filters to apply to your window rules (assuming you know AHK). But, for the casual user, it simply grabs the active window information and allows you to alter the conditions that apply to the window matching before saving those settings.

Downloadfrom GitHub: Muchtall Window Arranger
You must have AutoHotKey installed before using this script.

There’s one setting in the source you’ll probably want to modify before you get started. The variable “NumberOfScreens = 3” should be changed to the number of heads you want to have activate the auto-rearrange feature. If this is set to something higher than you’d ever have installed (say 10), the auto-rearrange feature should never activate.

Task Tray Menu

Capture/Save window settings

I’ll probably clean up the rough edges as I get feedback on it. I know there may be some use for features like re-arranging for multiple head settings, so there’s work to be done.

Edit: Moved code to GitHub repo

Brewblog: Kickoff w/ Northern Brewer Nut Brown Ale

My wonderful wife got me a Northern Brewer Deluxe Brewing Starter Kit (Glass) a couple of Christmases ago, and I’ve been slowly expanding my “brewery” ever since. I’ve made maybe 6 batches of beer using the kit. One of which I royally messed up (NB Bavarian Hefeweizen) when I scorched the malt in my newly-keggle-fied boiling vessel (on high heat of course).

So far, here’s my inventory of equipment:

Northern Brewer Deluxe Starter Kit:
– 6 Gallon Primary Fermentor (Glass), Fermometer, Bung, Airlock, Blowoff Assembly
– 5 Gallon Secondary Fermentor (Glass), Fermometer, Bung, Airlock
– 6.5 Gallon Bottling Bucket, Bottling Spigot, Bottle Filler, Bottling Tubing
– Auto-Siphon, Siphon Tubing
– Beer Bottle Brush, Bottle Capper
– Carboy Brush

I also had a few pieces of equipment from my prior homebrewing adventures of years past:
– Another bottle-capper
– Another 5 gallon secondary glass fermentor
– Airlock (from a Mr. Beer kit)
– One of those common orange dual-vent carboy caps
– ~3 Gallon stainless steel kettle (w/glass lid that I interchange with the keggle below)

And a few new acquisitions and creations:
– 15 gallon boiling “keggle”, modified from an old stainless Miller keg (inherited)
– 10lb. CO2 canister w/ regulator
– 4 ball-lock corny kegs
– 1 set of ball lock inlet/outlet tubing, “foam-free” faucet, and connections (I can tap one keg at a time)
– 4 Perlick 545PC Flow Control Faucets (Xmas gift from Santa/Father-in-Law)
– ~7.x cu ft. Freezer (from grandmother-in-law)
– Johnson Controls A419 Temperature Controller (to convert freezer into kegerator)
– Bayou Classic SP10 High-Pressure Outdoor Gas Cooker
– ~20′ counter-flow chiller w/partial convolution (see:
– ~2′ stainless steel spoon
– 2 reusable hop bags
– Yet-to-be-finished ~56qt mash tun cooler (still have to cut slots in my manifold, otherwise done)

Anyhow, I expect to be posting updates on future brewing exploits, so to kick it off, here’s my 6th-ish brew: the Northern Brewer Nut Brown Ale (extract kit). This is video I took of the fermentation just a mere 24 hours after directly pitching dry yeast into the wort. Not bad I think, considering that Danstar recommends prepping the yeast before pitching it.


Brewed: 4/14
OG: 1.050
FG: 1.012
ABV: 4.9%
Kegged: 5/21

Quick Tip: Identifying Space Consumption in Linux via Command line

Firstly, let me strongly recommend JDiskReport if a GUI is available to you. It’s super easy to use and helps you quickly drill down into the disk and identify disk space usage, on any platform with Java support.

In lieu of that, if you have need to identify disk usage via the command line, run this set of commands:
find / -exec du -ks {} \; 2> /dev/null | sort -n | awk '{printf $1 "\t"; if (system("test -d \""$2"\"")) { print $2 } else { print $2 "/" } }' | tail -1000

This will spit out a list of the largest 1000 individual files and folder sums, sorted by size.

Depending on the size of the disk, this will take a while, as should be expected as it runs a “du” for each path it finds. Caching helps if/when you re-run this.