Tips, Tricks, and Notes on running RAID1 and RAID5 on XCP-NG: Part 2

This is a continuation of a series. See Part 1 here.

Now that I have the base system installed on a RAID1 array, along with a Local Storage Repo residing there, I want to create an array for “bulk” storage. I have 3 8TB disks I want to put into a RAID5 array. This will give me a 16TB block device I can use as a sort of home NAS. Since this is RAID5, we can more efficiently expand this storage just by adding another 8TB+ disk. Additionally, I can use LVM to effectively split this storage between NAS storage and another Local Storage Repo for larger VM disk allocations.

Prep the disks

Before we do anything, we need to adjust the device timeouts on EACH drive you have installed. It’s not uncommon for SATA to take a long time to recover from an error, and if this results in a controller reset, you can risk RAID array failure and data loss. For more info on this, I would recommend reading . This article mentions setting the timeout on the disks. In my case, even though the command to set the timeouts didn’t fail, it never seemed to work, as the timeout values could not be listed and verified. Instead, to be safe, I’d recommend applying the 180 second (3 minute) timeout to all the drives you have in your system using this command:

for disk in `ls /sys/block/*/device/timeout` ; do echo 180 > $disk ; done

Then add a udev rule so that it applies on startup AND when new drives are inserted:

cat << EOF > /etc/udev/rules.d/60-set-disk-timeout.rules.test

# Set newly inserted disk I/O timeout to 3 minutes
ACTION=="add", SUBSYSTEMS=="scsi", DRIVERS=="sd", ATTRS{timeout}=="?*", RUN+="/bin/sh -c 'echo 180 >/sys$DEVPATH/timeout'"

systemctl restart systemd-udevd

Next, I would recommend removing all partitions on each disk and re-setting the disk labels to GPT. Beware! This and pretty much everything to follow is destructive. Be sure you are working with the correct disk! Checking the “dmesg” command output after inserting a new disk is usually how I verify.

parted /dev/sdz

# Respond with Y to confirm all data on the disk will be lost

Partition the drives (replacing /dev/sdz with your disk) . Repeat this for each of your disks:

echo -en "mklabel gpt\nmkpart primary ext4 0% 100%\nset 1 raid on\nalign-check optimal 1\nprint\nquit\n" | parted -a optimal /dev/sdz

RAID5 and LVM Setup

Now, let’s build our RAID5 array:

mdadm --create --verbose /dev/md6 --level=5 --raid-devices=3 /dev/sdx /dev/sdy /dev/sdz

# Set up mdmonitor mail
# In /etc/ssmtp/ssmtp.conf:

# In /etc/mdadm.conf:

Then partition the resulting md device to have one large LVM partition

gdisk /dev/md6

# Commands within gdisk...
# You should see no partitions

# Hit Enter (3x) when asked for partition number, first, and last sectors
# Use a Hex code of 8e00 for the filesystem type of Linux LVM
# You should then see something like this as your partition table, with a different size, obviously:
Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048     31255572446   14.6 TiB    8E00  Linux LVM

# Write partition table and exit gdisk:

# Answer "Y" to confirmation to write GPT data

To clarify the above, here are the commands that are run in gdisk:

(p)rint partition tables
(n)ew partition
(enter) Use partition number 1 (default)
(enter) Start at first sector on device (default)
(enter) End at last sector on device (default) 
(8e00) Linux LVM partition code
(p)rint the partition table
(w)rite the partition table and exit 

Next, let’s set up LVM using the partition:

# Tell the kernel to reload the partition table
partprobe /dev/md6

# Create the Physical Volume
pvcreate --config global{metadata_read_only=0} /dev/md6p1

# Create the Volume Group "BulkStorage00"
vgcreate --config global{metadata_read_only=0} BulkStorage00 /dev/md6p1

# Set the volume groups active so they come back up on reboot
vgchange --config global{metadata_read_only=0} -ay

### ??? Edit /etc/grub.cfg and remove "nolvm" from the "XCP-ng" menu entry

# Not sure if this is needed, but doesn't hurt (rebuild init ram fs)
dracut --force

Activating Volume Groups on Boot

Disclaimer: I know, I know. This is admittedly a hack. There’s probably a better way of doing this that fits within the design of XCP-NG that I’m not aware of. If someone suggests a better way of handling this, I’ll gladly update here.

When we need to reboot, our Logical Volumes won’t be active. I believe this has something to do with XCP-NG/XenServer’s unique handling of LVM. I’ve banged my head on this problem for far too long, so begrudgingly, here’s the workaround to it:

# Make rc.local executable
chmod u+x /etc/rc.local

# Enable the rc.local service
systemctl enable rc-local

# Add our vgchange activation command to rc.local
echo "vgchange --config global{metadata_read_only=0} -ay ; sleep 1" >> /etc/rc.d/rc.local

Local Storage Repository

Now let’s create a 200GB logical volume (start small as we can expand later)

lvcreate --config global{metadata_read_only=0} -L 200G -n "RAID_5_Storage_Repo" BulkStorage00

And create the Storage Repo in XCP-NG (using the “ext” filesystem for thin provisioning)

xe sr-create content-type=user device-config:device=/dev/disk/by-id/dm-name-BulkStorage00-RAID_5_Storage_Repo host-uuid=<tab_to_autocomplete_your_host_uuid> name-label="RAID 5 Storage Repo" shared=false type=ext

And that’s it! The above creates another set of LVM layers on top of your existing BulkStorage00/RAID_5_Storage_Repo logical volume which consists of a physical volume, volume group, and logical volume that are natively managed by XCP-NG. You should now have a 200GB storage repo named “RAID 5 Storage Repo” in XCP-ng Center and/or XOA. Next, let’s set up a bulk store to share out via SMB/CIFS…

Bulk Storage Share Setup

First let’s allocate 500GB of that space for another Logical Volume named “BulkVolume00”

lvcreate --config global{metadata_read_only=0} -L 500G -n "BulkVolume00" BulkStorage00

And lets format it with EXT4

mkfs.ext4 /dev/BulkStorage00/BulkVolume00

Let’s set up /etc/fstab (edit with vi or nano) to mount the device by adding the following line to the end of the file

/dev/BulkStorage00/BulkVolume00 /opt/BulkVolume00       ext4    rw,noauto        0 0

And lets create a mount point for the device and mount it

mkdir /opt/BulkVolume00
mount /opt/BulkVolume00

We’ll also want to set this up to mount on boot (note, as above with our Storage Repository, we have to use a workaround to mount this AFTER the logical volumes have been activated

echo "mount /opt/BulkVolume00" >> /etc/rc.d/rc.local

And lastly, lets get that shared out via SMB

# Install Samba server
yum --enablerepo base,updates install samba

# Make a backup copy of the original SMB configuration (just in case)
cp -rp /etc/samba/smb.conf /etc/samba/smb.conf.orig

Then edit /etc/samba/smb.conf, erasing the contents and using this as your template:

        workgroup = MYGROUP
        # This must be less than 15 characters.
        # Run "testparm" after editing this file to verify.
        netbios name = XCP-NG-WHATEVER
        security = user
        map to guest = Bad User

	comment = Bulk RAID5 Storage Volume
	path = /opt/BulkVolume00
	guest ok = yes
	writable = yes
        read only = no
	browseable = yes
        force user = root
	create mode = 0777
	directory mode = 0777
	force create mode = 0777
	force directory mode = 0777

Note that this is set up to be very permissive for use on a secured/home network. Setting this up for more elaborate security is beyond the scope of this tutorial.

Next let’s (re)start samba, and poke the required holes in the firewall

# Verify your smb.conf is sane
# Enable and start samba services
systemctl enable smb.service
systemctl enable nmb.service
systemctl start smb.service
systemctl start nmb.service

# Edit "/etc/sysconfig/iptables" and add the following lines below the port 443 rule:
-A RH-Firewall-1-INPUT -m conntrack --ctstate NEW -m udp -p udp --dport 137 -j ACCEPT
-A RH-Firewall-1-INPUT -m conntrack --ctstate NEW -m udp -p udp --dport 138 -j ACCEPT
-A RH-Firewall-1-INPUT -m conntrack --ctstate NEW -m tcp -p tcp --dport 139 -j ACCEPT
-A RH-Firewall-1-INPUT -m conntrack --ctstate NEW -m tcp -p tcp --dport 445 -j ACCEPT

# Then restart iptables to apply
systemctl restart iptables

Finally, test this share in Windows (or your favorite SMB/CIFS client) by opening the URI to your server’s IP, ex: \\\BulkVolume00\

The next installment of this series will be a rough collection of tips on managing this XCP-NG RAID5 combo. Stay tuned.

Disabling the Windows 10 “This app is preventing you from restarting/shutting down/logging off” screen

Sometimes Microsoft befuddles me. Scratch that. Not sometimes. Usually.

Take for example the “new” log-off/shutdown/restart behavior. Used to be by default in Windows 7 that when you shut down, you’d receive prompts about applications that were preventing shutdown. In most cases, this would be an unsaved document. You could simply click desired option (Save, or Don’t Save), and Windows would continue logging you out, unless it came upon another unsaved document. Rinse and repeat until you’ve closed all your unsaved documents, and Windows would finally log you out.

Now in Windows 10, you get that generic blue screen with a list of apps left to shut down, with only two options: Shutdown anyway, or Cancel. Neither of these are good options. One stops your shutdown, and you have to manually go through all your open apps and close them one by one. The other simple force closes all your apps, losing anything you might not have saved.

Thankfully, this setting is easily reverted to the legacy behavior with either a registry entry change, or a Group Policy change (which affects the same registry entry). To apply, use this reg file: Enable-AllowBlockingAppsAtShutdown.reg

Windows Registry Editor Version 5.00


You can also find this setting in the Group Policy editor (Start, Run: gpedit.msc) under Computer Configuration > Administrative Templates > System > Shutdown Options > Turn off automatic termination of applications that block or cancel shutdown:

Override Windows Explorer Win Key Hotkeys

For years now I’ve been using an awesome clipboard manager utility by JoeJoe Soft called ArsClip. One of the nice features of the program is the ability to set a custom hotkey for the clipboard manager clip list. In my case, I like to use the Ctrl+Win+V combo to invoke this feature. This can be set in the ArsClip INI file with these variables set:



There’s one problem though: Apparently Microsoft reserves the use of the Win key. If, as in my case, you start using Win+Something and it works for a while, a Windows 10 update could suddenly replace your cool clipboard functionality with something like, say, their own lame clipboard manager, or in my case, something completely useless and dumb like “Shoulder Taps“.

I suspect that there may be a clean way of having AutoHotKey take over these hotkeys and re-route them to ArsClip (or whatever application you want), however I couldn’t figure out how to make that happen.

There is a solution however. If you can start your app before the Explorer shell gets a chance to, then you effectively reserve those hotkeys for your app, and Explorer can’t use them. For a while, I’d do this by killing the explorer.exe process, run ArsClip, then re-start Explorer. Not ideal, but workable. I eventually came up with a better solution: Fire up ArsClip upon login, before Explorer starts, by using the Userinit registry key and a batch file to manage the timing. Here’s the batch file (C:\USERINIT.BAT):

@echo off
REM Start ArsClip and continue
start "" "C:\Program Files (x86)\ArsClip\ArsClip.exe"
REM Wait 2 seconds for ArsClip to start up and register hotkeys
timeout /t 2
REM Start the Explorer shell via UserInit.exe

Save that to C:\USERINIT.BAT
Then make this Registry modification (or save this to a .reg file and run):

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon]

Now when you log off and back on, ArsClip will start before Explorer, and your Win+Ctrl+V hotkey will work! Dirty hack? Maybe. If you know a better solution (such as some AHK magic as mentioned above) I’d love to hear about it.

Note that this modification will apply to all users of your computer. If you needed it to apply to just yourself, you may need to wrap it in some batch file logic, like so (untested):

@echo off
if '%username%' == 'myusername' goto hotkeyapps
goto userinit

start "" "C:\Program Files (x86)\ArsClip\ArsClip.exe"
timeout /t 2
goto userinit


Good luck!

Tips, Tricks, and Notes on running RAID1 and RAID5 on XCP-NG: Part 1

Note: This article series is specifically written with XCP-NG in mind, however if you are using XenServer, it is possible that some if not all of this is still applicable. YMMV.

Recently when installing XCP-NG on my home server, I ran into some persistent issues with setting up RAID1 on my boot disks, and setting up RAID5 for my secondary/bulk Local Storage Repository. Here are my notes and tips on succeeding with an installation such as this.

I have a chassis with 2x400GB SATA disks in a RAID1 array used for the boot disks that hold the OS and the first Local Storage Repository, and 3 8TB SATA disks in a RAID5 array with LVM on top, serving as both a Local Storage Repository as well as a bulk storage served via SMB/CIFS/NFS as a sort of NAS.

Installation Prep

Starting from scratch, we might want to wipe all the partition tables, boot records, and/or RAID superblocks on each target disk. This is optional, but if you know you don’t care about the data on your disks, it should help ensure success in case you had a prior installation of a RAID superblock or the GRUB bootloader on any of the disks. If you care about the data on any of the disks in your system, power off, unplug those disks, and start over. Boot up the installation disk, and type Alt+F2 to get to the console, then run the following:


for dev in `ls /dev/md*` ; do mdadmin --stop $dev ; done

for dev in `ls /dev/sd[a-z]+` ; do echo "Wiping Partitions and MBR on $dev" ; dd if=/dev/zero of=$dev bs=512 count=1 ; done

Next, reboot to start the installer over again, ensuring we start with a clean slate. Run the following to verify:


You should see no partitions or raid labels (ie, md127) on any of your target disks. If you do, you may need to re-run the above and try again.


Proceed as normal with the installation prompts, selecting the Software RAID option when it comes up.

Select the two (or more) disks you wish to add to the RAID1 array, then enter Create

Select your RAID disk as the install target (usually md127)

Select your RAID disk for your local storage (usually md127)

Proceed as normal with installation

When the install starts, type Alt+F2 to go to the CLI console

Run this to see the RAID resync process:

while true ; do tput clear ; date ; mdadmin --detail /dev/md127 | grep -v ^$ | grep -e ^ -e "Resync Status.*"; sleep 10 ; done

You should see a line starting with “Resync Status” indicating the percentage complete.

You may type Alt+F1 to go back to the installation progress screen.

Once the installation is complete, DO NOT REBOOT YET! I suspect this was the cause of an issue I had on one of my attempts doing this. The RAID1 array needs to finish syncing, or else you may be missing the required MBR/GRUB information on one of your disks, and your system may fail to boot if the non-synced disk happens to be the first in your BIOS/HBA boot order. Type Alt+F2 to go back to the CLI console and re-check the Resync Status progress. Once it reaches 100% complete and is in a State of “clean”, you may proceed.

Type Alt+F1 and finish the installation as normal.

Continue on with Part 2 in this series

A new era, a new look

Well, the Atathualpa theme has served me well since I first migrated my blog to WordPress from Slashcode (yeah, crazy). However we live in a new world now, with a growing majority of web traffic coming from mobile devices. It’s time I got with the times and used a responsive design theme. So I’ve made a switch to the TwentyFifteen theme. I’m going minimalist for now, and will likely bring modifications back in as I see value.

Jetpack Broke My Comments


So apparently “JetPack Comments” broke my comments, causing recent comments to get posted to the wrong thread. They claim it’s not their fault, saying that it’s due to a lack of implementation of the comment_form() function in WordPress, but Atahualpa seems to support this just fine. Well, anyhow, for now I’ll be disabling JetPack Comments on my blog. So if you want to comment, you’ll have to sign up. I know. It’s a pain. I’ll get it fixed.

Update 6/10/2015: After switching themes to TwentyFifteen, the problem seems to be gone. JetPack Comments re-enabled.

Adfree Breaks Pinterest on Android

If you’re like me, you like keeping your Android device screen free from ads. AdFree from BigTinCan is an invaluable tool in assisting with this by customizing your hosts file on a rooted Android phone so that any ad network links get redirected to your phone, effectively disabling ads. The side effect of using host-based ad blocking is that sometimes valid sites get blocked as well.

“I don’t always do Pinterest, but when I do, I prefer pinning homebrew stuff.” And unfortunately, Adfree blocks pinning on Android. You’ll notice this when doing any pin outside of a re-pin (ie, within your pin feed). The app will churn saying it’s finding images, but then finally fail with the popup error “Sorry, couldn’t find any pinnable images on this page”. The issue is that the Pinterest app requires access to a few hostnames that Adfree hijacks:


This solution to this is fairly simple. Thankfully, BigTinCan offers an option to set up a customizable exception list but of course you’ll have to register for a free account. Once you have registered, add exceptions for each of the hosts above. Then sign in to your account on the AdFree android app and update your hosts. You should now be able to pin to your heart’s content.

Let me know if this helped you!

Pebble Smartwatch Skins

One of my biggest beefs with the Pebble is the plastic case. These guys have a solution to not only the scratchability of the case, but the plain black style. The woodgrain one looks really nice, and is probably the one I would go with if I were to get a Pebble.

AHK Script to Clean Up Dead PuTTY Sessions

I have a love-hate relationship with tabbed interfaces. On one hand, they keep my taskbar clean. On the other, they merely scuttle my window-hoarding behaviors into a single window, where the mess grows into a pile of outdated, unused tab sessions (i.e., Firefox). I also have need for some types of programs to be opened in separate windows on a regular basis, such as SSH sessions, where I often work on multiple related tasks which have to be monitored simultaneously. In those instances, the number of PuTTY sessions I have open (and have disconnected) can quickly grow out of control, clogging my taskbar.

PuTTY Fatal Error

Here’s a quick script I wrote to clean up any inactive/disconnected PuTTY windows: Cleanup PuTTY Windows

Fix for Chromecast “No Cast devices found”

Every so often I run into a Chromecast user who says they can’t get their PC to find their Chromecast. More often than not, it’s due to a bug in how the Chromecast plugin handles multiple network connections. The Chromecast plugin assumes that your primary active network connection is on the same LAN as your Chromecast dongle. Obviously this is not always the case, especially in the case of VPN users. What you have to do is move that connection to the top priority in your network connection list.

Here’s how you fix the dreaded “No Cast devices found” issue. This is for Windows 7, but probably works with 8 and Vista as well:
1) If you are connected to a VPN, disconnect it and try detecting the Chromecast again. Still not working? Proceed…
2) Go to the Network and Sharing Center
3) Click “Change Adapter Settings”
4) Click the Advanced menu and select “Advanced Settings…” (Alt+N, S)
5) Find the network connection that shares the same LAN as your Chromecast and select it
6) Repeatedly click the green up arrow until that connection is at the top of the list.
7) Click OK to close the dialog and apply the new connection priority settings
8) Try detecting your Chromecast again

If you use a VPN, you may be wondering why you have to disconnect. The answer is that many VPN clients manage the connection priority of the virtual adapter they create and automatically moves it to the top of the list. It’s best just to disconnect and try again instead of battling that behavior.

Good Luck! I hope this helps someone out there!

If you found this helpful, maybe you’d like to send a thank you from my wishlist?

Brewblog: All-Grain Bavarian Hefeweisen (Northern Brewer)

Just a quick update on my homebrewing adventures.

Finally got around to doing my first AG brew using my new mash tun, using the Northern Brewer Bavarian Hefeweisen kit. The crushed grain sat in my garage for a couple weeks, then got tossed into the kegerator for storage for the next 4-5 months. Hopefully the grains didn’t go stale and affect the flavor of the malt. As this was my first AG experience, I had a slight hurdle with the volume of the strike (?) water. I was attempting to do the more complex multi-step protein/saccharide rests, and may have started out with far too much water. Thus I ended up with about 9 gallons of liquid that needed to be boiled down to ~5. I still don’t fully understand what grain/water  target ratio I should be using at mash-in. Needless to say, I burned off a good amount of propane just boling off those 4 gallons of excess liquid.

I’m unsure of how to properly calculate mash tun efficiency, however what I do know is that the listen target for original gravity was 1.049, and I ended up with 1.046. I was hoping for higher, as I did rinse the grain pretty well (or so I thought), and by comparison, my experience with the extract version of this kit has resulted in an OG of as high as 1.052. The comparison might not be valid though.

Fermentation was very active, as usual, finishing up it’s frothy activity within the first 36-48 hours. This time around I didn’t have any issues with blowoff.

So, with any luck, I should have a perfectly drinkable beverage in just under 4 weeks. Now to figure out where to re-fill my 10# CO2 cylinder.

Here’s my current brewhouse/pub status:

On deck:        Pumpkin Ale (Indie)
Primary:        Bavarian Hefeweisen (Northern Brewer)
Secondary 1:    Empty
Secondary 2:    Empty
Keg 1:          Tapped-out
Keg 2:          Tapped-out
Keg 3:          Tapped-out
Keg 4:          Tapped-out

HowTo Resolve StartSSL (StartCom) Domain Blacklisted: Domain appears on a blacklist

Does this look familiar to you?


Welcome to my world. Not sure at this point how I got on this list, how to get off it, or even where this list is. But perhaps my findings will help you resolve the same issue for your domain. At this point, my suspicion is that it’s due to and odd report from Google Safe Browsing that “Yes, this site has hosted malicious software over the past 90 days. It infected 0 domain(s), including .”. It would be great if I knew what the malware/badware is/was so that I could remove it. Even more odd is that my supposed infection infected no other sites.

Oh well. More to come…

Update: I’ve emailed “Certmaster” and they responded letting me know that they see my domain on Google’s Safe Browsing blacklist results. Oddly enough, here are the results:

I see the report that “Yes, this site has hosted malicious software over the past 90 days. It infected 0 domain(s), including .” What’s odd about this is that when I check my Google Webmaster tools, the site reports that “Google has not detected any malware on this site.”, and it seems I’m not the only one. Not sure if I’m just bitten by a previously unseen issue that I’ve since cleaned up with WordPress updates or what.

Given the date that is shown above (2013-05-04), I’d guess that the “past 90 days” implies I’ll have to wait until 2013-08-04 for this status to clear. I guess that’s the penalty I pay for lack of diligence in monitoring the updates and health of my server up until then. If you’re saying to yourself “I can’t wait that long!”, you do have the option of paying StartSSL the fee required for them to manually intervene in what would otherwise be an automated process. I choose to wait it out: I don’t really need SSL for anything practical. For my purposes, it’s just for the sake of writing articles like this: research and writing howto’s based upon my experiences. So I’ll be waiting out the presumably prerequisite “90 days” for the sake of research.

See you on 8/4 with an update!

Update 8/15: As the saying goes: “Time heals all wounds”. I’m now off the naughty list for Google. Now to (re-)try obtaining a cert from StartSSL…

Accidental Evangelist

It’s been a while since updating my blog with anything overtly Catholic. I guess work, family, and life in general started to consume far more of my time in last several years. Non-geek-related posts have been sparse to non-existent. It’s been bothering me some privately in recent months with elections coming, and now gone. There’s so much topical matter to cover, and unfortunately, most of it gets posted to Facebook where my audience is limited (and intentionally so).

If been feeling that, as a result, I haven’t been doing a very good job of being an active evangelist for Catholicism. It’s been taking a back seat to being charitable with my technical knowledge. I’ve gained a lot of knowledge from peers in the past 15+ years, and this blog is one of the ways I like to “pay forward” that generosity.

Which brings me to this blog article by Creative Minority Report: Accidental Evangelism.

I think my sense of charity directly, and most specifically a personal impressibility to be actively charitable stems from my Catholic upbringing. I’d could only hope that some person reading this blog would be converted (at least in part) by my actions.

May God bless, and the grace of God be with you!

Solution to the NVidia Gray/Grayscale Screen Problem

Today I came in to work to find that the video output for my Dell Latitude e6520 laptop’s NVidia head was displaying in black and white. At first I thought that the problem was a driver bug, something wrong with the video memory, or a faulty display. But eventually I found out that, somehow (without any user interaction with the applicable setting), the “Digital Vibrance” setting was set to 0%, when it should be 50%. Below you’ll see a simple annotated screenshot showing where you can quickly fix this.

Nvidia Gray Screen Problem - Annotated

Good luck!

Streaming RTMP with VLC and RTMPDump

This quick post is as much for your benefit as for the benefit of my memory…

To stream RTMP with VLC, you’ll need rtmpdump, which you can get here: I used, though you may be able to use the latest version. I also had VLC 2.0.6 32-bit installed. Once installed, you can run the following from a cmd window:

rtmpdump.exe -r “rtmp://” -v -o – | vlc.exe –

This worked nicely for me. YMMV. Good luck!

If you found this helpful, maybe you’d like to send a thank you from my wishlist?

FFMPEG “Server error: Not Found” with Short URLs

Just a quick post about a problem I helped a buddy of mine resolve. He was setting up a Helix media streaming server, and was trying to capture the stream data to a file with the following command:

ffmpeg -i “rtmp://” out.flv

The result was this error in the output:

[rtmp @ 0x28e1dc0] Server error: Not Found

Oddly enough, the connection information shown on the Helix console showed that a strange URL was being requested. Upon further investigation with Wireshark, I found that this was the request being made.


Note that “\360” is a hex character. For some odd reason, it would appear that ffmpeg improperly handles short URLs, inserting a string of “\360xw0”. If you pad the URL with the current directory “./”, then the request succeeds:

ffmpeg -i “rtmp://” out.flv

This results in a request of


Which worked fine in our environment.

For future reference, I was running this ffmpeg version (on CentOS 6.4 x86_64):

ffmpeg version N-53616-g7a2edcf Copyright (c) 2000-2013 the FFmpeg developers
built on May 29 2013 00:19:54 with gcc 4.4.7 (GCC) 20120313 (Red Hat 4.4.7-3)

So if you’re running into the “Server error: Not Found” error on a known good URL, try padding the path of the stream with “./” and see if that fixes it for you. I’m guessing this is an ffmpeg bug, but don’t really have the access to a streaming server to troubleshoot and submit a bug report. From the time that I did have, it appears that it’s related to the .flv extension in the rtmp URL. If you drop the extension, the URL can be of any size.

Brewblog: Rubbermaid 50 Quart Mash Tun

After several months of lingering in my garage, I finally finished my 50 qt. Rubbermaid mash tun. The cooler I used is the one shown below:



I picked it up at my local Menards for about $30. You should be also able to find them at Walmart for about $40. I chose this cooler because I’ve purchased a few of them over the past years, so I know they’ll likely be making them for a while. Also, the design is relatively easy to build a mash tun manifold for. And, according to this chart, I should be able to make up to 10 gallons of wort comfortably in there, allowing me to eventually make double batches.

The manifold is a design that I came up with, and eventually I’ll get around to actually measuring and posting dimensions of the piping so you can build one too. I used under 8′ of copper pipe for it, and should have enough length left over to eventually build a fly sparge manifold (or at least a start on it). The design relies on two lengths of  bare copper wire to hold the whole thing together, and two small segments of tubing at the back of the cooler keeps the manifold  securely wedged in position. I’m not averse to soldering, or even all that concerned about contamination from the silver-based solder. My main reason for going solderless is to make it easy to clean. I can literally disassemble the entire manifold into individual components, ready for a thorough brushing if desired.


Recovering the PPP Username and Password from a Centurylink Actiontec C1000A

Some time ago I wrote up a similar procedure to recover a password from an Actiontec M1000 back when Centurylink was known as Qwest (gotta love rebranding). Back then, Actiontec left the operating system a bit more open, actually placing the PPP credentials in a flat file in /var/tmp/. Nowdays, Actiontec tries to obfuscate/encrypt the password in a config XML, making it just difficult enough for most people to give up on the idea of recovering the password.

In addition to this, it would appear to the casual telnet console user that the commonplace busybox shell had been removed or made inaccessible, removing the ability to peer into the embedded linux operating system underneath, and replacing it with a stripped-down properietary shell with limited commands. However, dig a little deeper by trying the undocumented “sh” command, and you’ll find that busybox is alive and well on this device, exposing the configs and services that support the router’s functions. Since the configs contain the encrypted password, we can’t directly extract them from there, however fortunately for us, pppd (the service that authenticates and creates the DSL connection) requires the password to either be kept in plain text in a flat config file (not the case here) OR have the password specified on the command line. The “ps” command on these devices has been handicapped to only display 80 columns, leading to output similar to this:

1623 admin      1144 S   pppd -c ppp0.1 -D 0 -i ptm0.0 -u “your___username@qwe

At first, when I saw this, I figured it was another dead end, until I realized that “/proc/(pid)/cmdline” displays the command line of any running process. And fortunately for us, Actiontec left “pidstat” enabled in busybox, making reading those command line arguments fairly simple with a single command:

 > sh -c “/usr/bin/pidstat -l -C pppd”
Linux 2.6.30 ((none))   05/23/13        _mips_  (2 CPU)

14:59:38          PID    %usr %system  %guest    %CPU   CPU  Command
14:59:38         1623    0.00    0.01    0.00    0.01     1  pppd -c ppp0.1 -D 0 -i ptm0.0 -u “” -p “AbCDEfgH” -f 0 -k -P “AbCDEfgH” -M 1492

And there you have it! In this case, I’ve altered the password output (shown as AbCDEfgH) to protect my own privacy, but it’ll look similarly like jumbled letters and numbers on your command line. Also note the “” username. Qwest used to be the LEC in my area, and likely the transition hasn’t been made to the newer branding in the back-end systems. Your situation may be different.

Also, in case you hadn’t figured it out by now, you’re going to need telnet console access. My modem’s console admin password was not the default “admin”, nor the admin password listed on the box. I had to log into the web UI, enable remote console, and (re)set the console password. After that, you should be able to log into the telnet console with “admin” and the password you’ve set.

Lastly, one might ask “Why would I want to obtain this password?”. My personal reason is that eventually I want to replace this leased modem with one I’ve purchased, and use it in transparent bridging mode (using RP-PPPOE to terminate the PPPoE connection and it’s leased static IP on my firewall). When that day comes, I’d like to be able to do the swap without interacting with Centurylink. Call them by whatever new brand they’ve been changed to, Ma’ Bell is always a pain to talk to.

Good luck to you!

If you found this helpful, maybe you’d like to send a thank you from my wishlist?

Pitfall-Free Howto/Guide to StartCom/StartSSL Class 2 Organization Validation/Certification

…in just three not-so-simple steps.

A couple quick notes before we begin: I threw this article together over a period of weeks, so the layout is a bit… odd. At some point I’ll come back to this article and clean it up, however for now, I think it does the job of conveying the process to a new StartSSL user. Also, I use the names “StartCom” and “StartSSL” interchangeably, so don’t look any deeper into the name usage than that.

A few weeks ago, I started the process of renewing several StartCom certs for my employer, and started to become familiar with the processes and pitfalls of identity and organization validation with StartCom. After delving into the process head-first, it became evident that this might not be as straight-forward as one might expect. However, I think that once you know what to expect, the process should go much more smoothly.

Also, please note that this how-to merely documents what I did to get a cert, and what pitfalls I ran into along the way. Therefore, your requirements and path may (and probably will) vary from mine. For instance, the tax document I submitted was a “State of Delaware Annual Franchise Tax Report”. This document lists all of the items shown on StartCom’s requirements. Your document(s) may not.

First, let’s understand the steps required in obtaining Class 2 Org validation. Basically, you keep escalating your level of validation, providing increasingly trusted levels of personal and organization documentation as you go. Here’s a rough outline:

  • Get Email Address validation with StartSSL (free)
    • Sign up
    • Validate email
    • Get personal client cert (for browser) via website
      • Back it up!!!
  • Get Personal Class 2 validation (~$60)
    • Submit 2 forms of ID
      • 1 Photo ID (i.e.: Driver’s license)
      • 1 other form of ID. May be photo (i.e.: Passport), OR non-photo (i.e.: birth certificate)
    • AND submit either:
      • Phone bill showing your name, current address, AND (most importantly) phone number.
        • May be land line or cell


      • Request for certified mail validation
        • Delivery will take ~2.5 to 5 weeks (from Israel)
          • If expedited service is required, you may additionally request express mail (~4-5 bus. days) for approx $30.
    • Wait for call or letter, and verify provided code on StartSSL website control panel.
  • Get Organization Class 2 validation (~$60)
    • Submit tax document which contains:
      • Name of CEO
      • Names of Directors
      • Co. Address (not sure if this is an actual requirement)
      • Co. Phone number (not sure if this is an actual requirement)
    • AND submit StartSSL’s “Delegate Authorization Letter” ( with:
      • Your name, title as delegate
      • Name and signature of CEO, President, or Director (CFO also appears to work. Other C?? titles may suffice)
    • Have your signatory receive a phone call from StartCom to verify your authorization.
  • Validate your domain(s)
    • Perform the “Domain Name Validation” validation wizard for each new or expiring domain
    • Confirm validation using code sent to email
  • Generate Certs
    • Perform the “Web Server SSL / TLS Certificate” certificate wizard
      • Only new/expired certs or certs expiring within 2 weeks may be renewed (when the existing cert is with StartCom)
    • Generate password-protected key / cert pairs
      • Keep that password safe and backed up!
    • Save key to .key file, cert to .crt file (ie,,
  • Install Certs
    • Decrypt (strip password protection from) key file
    • Place decrypted key and crt files on webserver
    • Configure Apache SSL
    • Reload Apache
    • Verify correct cert in web browser
  • ???
  • PROFIT!!!

And now for the more elaborate explanation:

1) Get your email address set up, and obtain a website authentication certificate. This is free. Go to the Control Panel and click Sign Up. Fill out the forms and submit. Soon you should receive an authentication code via email. You will submit this to the StartCom site, and then wait for another email that has a verification link, which you’ll click. Once that’s done, you’ll get a personal client certificate for your browser. Follow the instructions in the email/website on how to backup this certificate. Don’t skip this! Think of this as cert the key to your account. If you lose it, you’re likely going to have to a hard time regaining access. Back it up to an external storage medium (perhaps even encrypted as well – TrueCrypt is good for this).

2) Get yourself a Class 2 validation. This will cost you ~$60 (with one potential caveat, described below). This is where things start to get more complex. You’ll need two scanned or photographed forms of documentation that prove that you exist, AND a means of verifying that the person submitting the documentation is actually you. For the two forms of identification, one must be a photo ID (afaik), and another can be another photo ID (such as a passport) OR something that simply proves you exist, such as a birth certificate (this is what I used). Submitted photos/scans must be of “high” quality, but less than 1400×1400 resolution. The second half of this, and this is where I got hung up, is the verification that the person submitting the information is YOU. StartCom apparently trusts phone companies, because you “just” need to submit an invoice/bill showing your name, address, and phone number. This can be either a scan/photo or a PDF. This is where I got hung up. My current cell service is not in my name, and I have Ooma VOIP for home phone service, which doesn’t do traditional paper or PDF invoicing. Attempts to submit website screenshots, or PDFs of a webpage will likely be rejected. StartCom needs to be able to verify that you own the number you say you do, and then call you at that number to verify that you submitted the request (presumably with a code conveyed by voice, then submitted to the StartCom website). In lieu of a verified phone number, your next/only option is a registered mail letter from StartCom (in Israel). They claim this takes 3-5 business days. That may be true if you lived in Israel. I live in the US. Internet rumor has it that this letter takes ~5 weeks to get to the US destinations. I’ll see if one of my letters end up taking this long (see update below… took ~2 weeks). The alternative is to request express mail delivery, which costs ~$30. This will take approximately 4-5 business days. I opted for this to expedite the process. Once you receive the letter, it will contain a code which you submit via the StartCom website control panel. Shortly after, StartCom should send you confirmation that you have been personally Class 2 verified.

3) Get your Organization Class 2 verified (another ~$60). This part is still in progress for me. I will be updating as I go along. First step appears (at least for now) that I have to obtain a yet-to-be-identified tax document from our finance/accounting department that proves that our company is recognized to exist by either the State or the US. Also, it’s not clear if I, as an employee of the company, can submit the tax documentation and as a result get access to create these certs, OR do I also need to submit an authorization form that delegates me to make cert changes on behalf of the company. Sure would be nice if StartCom would clear some of this up on their website, or at least on their forums.

More on this as the mystery unfolds…

Update 5/28: A few updates

  • Today I received my initially-requested snail-mail letter from StartCom. The letter was requested late on the 10th, so that means it takes 14-16 postal days (Mon-Sat) to get to Minneapolis, MN. But of course, YMMV.
  • In the case of my company, the tax document used was a Delaware Annual Franchise Tax Report (as the company is incorporated in Delaware). Your document will almost surely be different. My recommendation is to ask your accountant or executive for a copy of the Articles of Incorporation applicable to your state.
  • Unless you are the CEO, President, or a Director of the company/org you are applying for, you’ll need to fill out the  StartSSL Delegate Authorization Letter, printed on company letterhead, and have one of the aforementioned individuals sign it. Currently I’m attempting to use our CFO’s signature to see if that is sufficient. If not, I’ll be going to CEO to get the required signature.

Update 6/4: Eventually I got a signature from our CFO. I submitted the document to StartCom, and within a few hours I got an email back they they were unable to reach anyone at the listed office number. Presumably they tried to contact our CFO. I asked them to try again, and soon after got an email stating that the Class 2 Org Validation is complete! Once that was done, I was able to log into the StartCom control panel and perform Domain Name Validation(s) for each of our domains. It appears they offer a number of hostmaster/postmaster addresses as options to send a validation email to your domain. In addition to our other TLD’s, they offered Not sure how they linked them to our main corporate domain (possibly public WHOIS data), but they did. Up next, cert generation!…

Update 6/7: This ended up being the most straightforward process of the entire adventure. In my case, the web service in use was Apache. This process will likely vary for an MS-based/IIS server. Once your domain(s) is/are validated, you can go back into the StartSSL control panel, go the the certificates wizard, and select “Web Server SSL/TLS Certificate”. Set a password for you keyfile and generate the key. Keep good tabs on this password as it’s what decrypts your private key for use on your webserver. Save the resulting key text out to a file named something like “”. Continue on and select your domain, and enter/add your subdomain(s) or subdomain wildcards as needed (see notes below regarding restrictions). Once the cert is generated, save the text into a file named something like “”. Keep these files safe and backed up!

A few other notes related to this: Note that a subdomain wildcard only applies to the level of that wildcard. So for instance, * would cover, byt not For that, you need to specify wildcards for each recursive level, ie * Also note that domains that still have active StartSSL certs (or are not expiring within 2 weeks) are not able to be renewed. If you must renew them, you must revoke the existing cert, which costs approx $25. Be aware that you could still generate certs for individual specific (sub)domains instead to avoid going through the revocation process.

You can now proceed to install the cert on your webserver, however note that hte .key file is password protected/encrypted. You must strip the password in order to allow your web service to start up unattended. Here’s a quick command to do that:

openssl rsa -in >

You can then take the decrypted key and use it in your Apache config. Please note however that you must take care to restrict access to your server so that your private key is not copied by unauthorized persons. They alternative to this is to leave password protection on, but intervene manually by entering in the password each time Apache is (re)started.
So there you have it! At some point I’ll probably come back to this article and clean it up. For now, I just wanted to get the information out there to help my fellow interweb users. Good luck!

Routed OpenVPN HOWTO

This is my OpenVPN HOWTO. There are many like it, but this one is mine.

It seems every few months I get asked the question by one of my friends “How do I set up a VPN?”. Sometimes the person is looking to set up a MS VPN variant, other times, OpenVPN. The principles and concepts seem simple to me, now, however for someone new to VPN architectures and perhaps even routing, it can be confusing. This is my attempt to make the mysterious understandable. Since roadwarrior (individual laptop clients) configs are fairly well documented by the official OpenVPN wiki, I’ll concentrate on a simple routed, LAN-to-LAN VPN networking concept, and cover roadwarrior config as an afterthought.

My weapon of choice distro-wise is CentOS, however these instructions could be applied to any other distro (ie, Ubuntu) with a basic understanding of your particular platform’s networking configuration methods. Really, OpenVPN can run on just about anything, including Windows, if you feel so inclined. However, you’ll probably get the most bang for your buck (free) using Linux. In my test environment, running stock CentOS 6.4, I had the scenario below running on a server with less than 90MB of total system memory usage.

Why CentOS?
I have two main reasons for promoting the use of CentOS:
1) CentOS is based upon RedHat Enterprise Linux (RHEL), which currently enjoys status as the most vendor-supported OS in enterprise environments. Translation? Knowledge of an RHEL derivitaves like CentOS is marketable skill to put on your resume. Yes, Ubuntu is gaining popularity in tech circles, but still doesn’t compare to RHEL for vendor support. There’s a reason both VMWare and Citrix use RHEL derivitaves as their baremetal OS.
2) CentOS has long-term-support (LTS). I’ve used Fedora for years, and I enjoy(ed) playing with some of the bleeding-edge features it offers. But the bleeding edge is on a double-edged sword. Fedora has a relatively aggressive release and support schedule. Install Fedora X, and expect that Fedora Z will replace it in about a year, leaving version X effectively without support. This gets to be a pain when you need to “yum update” your system just a year or so after you installed it. CentOS however has a support schedule that will ensure you likely have updates for far more years than the lifecycle of your hardware. For example CentOS 6, released in 2011, reaches EOL in late 2020. That’s almost 10 years of support, on a free platform!

First, let’s diagram the network we are going to design:

Remote Office 1 (10.101.0.x/24)
VPN Tunnel (
Main Office (10.100.0.x/24)
VPN Tunnel (
Remote Office 2 (10.102.0.x/24)

For this guide, we’re going to assume you want all remote offices to have routing enabled to each other (via the Main office).

Note that the IP addresses above are just for example. You could create your own IP addressing scheme with FAR better utilization of the the private address space. The subnets I have used (ie 10.200.x.x/16) are just for increased clarity.

Each OpenVPN server/endpoint can have one interface (assuming you are doing NAT/firewalling elsewhere on your network), OR you could have dual interfaces: One on the LAN, the other on the WAN/Internet connection. It’s up to you depending on where you want to do your firewalling. For the purposes of this guide, we’re putting the VPN server behind another firewall.

On your firewall
Forward UDP port 1194 for your external internet connection to the internal IP of your VPN servers/endpoints, ie:

Main Office
Remote Office 1
Remote Office 2

If for some reason UDP/1194 were blocked by your ISP, you could switch to something like TCP/80, but for the purposes of this guide, we’ll stick with the default UDP/1194.

On each server/endpoint
Install your base OS. I chose to just install CentOS 6.4 minimal installation with the default options. Once installed, get networking set up and run

yum update

and get everything up-to-date.

Install the EPEL Repo configs

yum install

Turn off the built-in firewall (remember, we’re using a 3rd party device for this). You could leave it on, but then you’d have to configure it to pass traffic to/from your VPN tunnel interface. I’ll leave that up to you to figure out should you choose to do so.

chkconfig iptables off ; service iptables stop

Install the OpenVPN packages and dependencies:

yum install openvpn

Server (Main Office)

Copy the the easy-rsa environment to /etc/openvpn/easy-rsa, do some config prep, and alter the vars file

cp -rp /usr/share/openvpn/easy-rsa/2.0 /etc/openvpn/easy-rsa
cd /etc/openvpn/easy-rsa/
cp -rp vars vars.orig
cp -rp openssl-1.0.0.cnf openssl.cnf
vi vars

Go down to “KEY_COUNTRY” and edit the Country, Province, etc, down to the OU, ie:

export KEY_CITY="Minneapolis"
export KEY_ORG="Muchtall"
export KEY_EMAIL=""
export KEY_NAME=MuchtallOpenVPNServer
export KEY_OU=Muchtall

Now generate your server’s certificate authority:

. ./vars

Accept the defaults for the prompts (we already set them)

Now build the server key


Similar to above, accept the default prompts. You will have to answer “y” to the questions “Sign the certificate? [y/n]:” and “1 out of 1 certificate requests certified, commit? [y/n]”

Now generate certs/keys for each remote site

./build-key remote-office-1
./build-key remote-office-2

Rinse, repeat on the prompts.

And generate the Diffie-Hellman parameters:


Great! Our certificates are all set up!

Next time you want to generate a new client key, just run

cd /etc/openvpn/easy-rsa/
. ./vars
./build-key remote-office-3

Now, let’s set up the configs. There’s a sample config at /usr/share/doc/openvpn-2.2.2/sample-config-files/server.conf, however we’re going to set one up using this template, just to keep things simple:

port 1194
proto udp
dev tun
ca easy-rsa/keys/ca.crt
cert easy-rsa/keys/
key easy-rsa/keys/
dh easy-rsa/keys/dh1024.pem
ifconfig-pool-persist ipp.txt
# Tell clients that we can handle routes for these networks
push "route"
push "route"
push "route"
client-config-dir ccd
# Tell OpenVPN that it routes for anything within these subnets
keepalive 10 120
status openvpn-status.log
verb 3

Now create the ccd directory

mkdir /etc/openvpn/ccd

And lets create the client-specific configs to route each individual subnet to the respective site:

# In /etc/openvpn/ccd/remote-office-1


# In /etc/openvpn/ccd/remote-office-2

Good? Good!

Normally, IP subnets for the tunnels are allocated as new tunnels connect to the server. Let’s pre-set the IPs for each tunnel. This part isn’t necessary, however I like to do this to assist with clarity in troubleshooting with traceroutes. In /etc/openvpn/ipp.txt:


And re-set the SELinux permissions on the ipp.txt file

restorecon -v './ipp.txt'

Now we’re ready to start the the OpenVPN service up:

service openvpn restart

Check the syslog to see if anything serious got spit out:

tail -100 /var/log/messages

You should see something similar to this:

May 7 14:54:53 mainoffice openvpn[13362]: OpenVPN 2.2.2 x86_64-redhat-linux-gnu [SSL] [LZO2] [EPOLL] [PKCS11] [eurephia] built on Aug 10 2012
May 7 14:54:53 mainoffice openvpn[13362]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
May 7 14:54:53 mainoffice openvpn[13362]: Diffie-Hellman initialized with 1024 bit key
May 7 14:54:53 mainoffice openvpn[13362]: TLS-Auth MTU parms [ L:1542 D:138 EF:38 EB:0 ET:0 EL:0 ]
May 7 14:54:53 mainoffice openvpn[13362]: Socket Buffers: R=[229376->131072] S=[229376->131072]
May 7 14:54:53 mainoffice openvpn[13362]: ROUTE default_gateway=
May 7 14:54:53 mainoffice openvpn[13362]: TUN/TAP device tun0 opened
May 7 14:54:53 mainoffice openvpn[13362]: TUN/TAP TX queue length set to 100
May 7 14:54:53 mainoffice openvpn[13362]: /sbin/ip link set dev tun0 up mtu 1500
May 7 14:54:53 mainoffice openvpn[13362]: /sbin/ip addr add dev tun0 local peer
May 7 14:54:53 mainoffice openvpn[13362]: /sbin/ip route add via
May 7 14:54:53 mainoffice openvpn[13362]: /sbin/ip route add via
May 7 14:54:53 mainoffice openvpn[13362]: /sbin/ip route add via
May 7 14:54:53 mainoffice openvpn[13362]: Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
May 7 14:54:53 mainoffice openvpn[13369]: UDPv4 link local (bound): [undef]:1194
May 7 14:54:53 mainoffice openvpn[13369]: UDPv4 link remote: [undef]
May 7 14:54:53 mainoffice openvpn[13369]: MULTI: multi_init called, r=256 v=256
May 7 14:54:53 mainoffice openvpn[13369]: IFCONFIG POOL: base= size=16382
May 7 14:54:53 mainoffice openvpn[13369]: IFCONFIG POOL LIST
May 7 14:54:53 mainoffice openvpn[13369]: remote-office-1,
May 7 14:54:53 mainoffice openvpn[13369]: remote-office-2,
May 7 14:54:53 mainoffice openvpn[13369]: Initialization Sequence Completed
May 7 14:54:53 mainoffice kernel: tun0: Disabled Privacy Extensions

Once you’ve verified based upon the above output that everything is running fine, go ahead and mark the service to start automatically

chkconfig openvpn on

Client Configs (Remote Offices)
On each server, create the file “/etc/openvpn/”, and populate with the following:

dev tun
proto udp
remote 1194
resolv-retry infinite
ca ca.crt
cert remote-office-1.crt
key remote-office-1.key
verb 3

Be sure to change the cert name as appropriate.

Copy the ca.crt, remote-office-1.key, and remote-office-1.crt to the /etc/openvpn/ directory of the client. Repeat for Office 2.

Set the permissions on the key file so that it can’t be copied by non-root users.

chmod 600 /etc/openvpn/remote-office-1.key

Start the OpenVPN service

service openvpn start

Check the output of syslog for similar output:

May 7 16:26:39 remote-office-1 openvpn[1566]: OpenVPN 2.2.2 x86_64-redhat-linux-gnu [SSL] [LZO2] [EPOLL] [PKCS11] [eurephia] built on Aug 10 2012
May 7 16:26:39 remote-office-1 openvpn[1566]: WARNING: No server certificate verification method has been enabled. See for more info.
May 7 16:26:39 remote-office-1 openvpn[1566]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
May 7 16:26:39 remote-office-1 openvpn[1566]: LZO compression initialized
May 7 16:26:39 remote-office-1 openvpn[1566]: Control Channel MTU parms [ L:1542 D:138 EF:38 EB:0 ET:0 EL:0 ]
May 7 16:26:39 remote-office-1 openvpn[1566]: Socket Buffers: R=[229376->131072] S=[229376->131072]
May 7 16:26:39 remote-office-1 openvpn[1566]: Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
May 7 16:26:39 remote-office-1 openvpn[1566]: Local Options hash (VER=V4): '41690919'
May 7 16:26:39 remote-office-1 openvpn[1566]: Expected Remote Options hash (VER=V4): '530fdded'
May 7 16:26:39 remote-office-1 openvpn[1567]: UDPv4 link local: [undef]
May 7 16:26:39 remote-office-1 openvpn[1567]: UDPv4 link remote: x.x.x.x:1194
May 7 16:26:39 remote-office-1 openvpn[1567]: TLS: Initial packet from x.x.x.x:1194, sid=f439995e ac9dd302
May 7 16:26:39 remote-office-1 openvpn[1567]: VERIFY OK: depth=1, /C=US/ST=MN/L=Minneapolis/O=Muchtall/OU=Muchtall/
May 7 16:26:39 remote-office-1 openvpn[1567]: VERIFY OK: depth=0, /C=US/ST=MN/L=Minneapolis/O=Muchtall/OU=Muchtall/
May 7 16:26:39 remote-office-1 openvpn[1567]: Data Channel Encrypt: Cipher 'BF-CBC' initialized with 128 bit key
May 7 16:26:39 remote-office-1 openvpn[1567]: Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
May 7 16:26:39 remote-office-1 openvpn[1567]: Data Channel Decrypt: Cipher 'BF-CBC' initialized with 128 bit key
May 7 16:26:39 remote-office-1 openvpn[1567]: Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
May 7 16:26:39 remote-office-1 openvpn[1567]: Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 1024 bit RSA
May 7 16:26:39 remote-office-1 openvpn[1567]: [] Peer Connection Initiated with
May 7 16:26:41 remote-office-1 openvpn[1567]: SENT CONTROL []: 'PUSH_REQUEST' (status=1)
May 7 16:26:41 remote-office-1 openvpn[1567]: PUSH: Received control message: 'PUSH_REPLY,route,route,route,route,topology net30,ping 10,ping-restart 120,ifconfig'
May 7 16:26:41 remote-office-1 openvpn[1567]: OPTIONS IMPORT: timers and/or timeouts modified
May 7 16:26:41 remote-office-1 openvpn[1567]: OPTIONS IMPORT: --ifconfig/up options modified
May 7 16:26:41 remote-office-1 openvpn[1567]: OPTIONS IMPORT: route options modified
May 7 16:26:41 remote-office-1 openvpn[1567]: ROUTE default_gateway=
May 7 16:26:41 remote-office-1 openvpn[1567]: TUN/TAP device tun0 opened
May 7 16:26:41 remote-office-1 openvpn[1567]: TUN/TAP TX queue length set to 100
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip link set dev tun0 up mtu 1500
May 7 16:26:41 remote-office-1 kernel: tun0: Disabled Privacy Extensions
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip addr add dev tun0 local peer
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip route add via
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip route add via
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip route add via
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip route add via
May 7 16:26:41 remote-office-1 openvpn[1567]: Initialization Sequence Completed

And verify that the routing is taking place


Set the openvpn service to start automatically

chkconfig openvpn on

Repeat these steps for the Office 2 client. And verify that you can ping across both tunnels to Office 1

Roadwarrior Config (this is optional)
As an afterthought, I said I’d cover Roadwarrior configuration. Here’s a basic rundown:

– Generate a new cert/key pair for your username using the above ./build-key commands
– Install OpenVPN for Windows (if you’re on Mac or Linux, you likely know how to do this already)
– Copy the ca.crt, username.crt, and username.key files to “C:\Program Files\OpenVPN\config\”
– Create a config file named C:\Program Files\OpenVPN\config\ with these contents:

dev tun
proto udp
remote 1194
resolv-retry infinite
ca ca.crt
cert myusername.crt
key myusername.key
verb 3

Right-click on the OpenVPN GUI in the taskbar and click “Connect”.