Routed OpenVPN HOWTO

This is my OpenVPN HOWTO. There are many like it, but this one is mine.

It seems every few months I get asked the question by one of my friends "How do I set up a VPN?". Sometimes the person is looking to set up a MS VPN variant, other times, OpenVPN. The principles and concepts seem simple to me, now, however for someone new to VPN architectures and perhaps even routing, it can be confusing. This is my attempt to make the mysterious understandable. Since roadwarrior (individual laptop clients) configs are fairly well documented by the official OpenVPN wiki, I'll concentrate on a simple routed, LAN-to-LAN VPN networking concept, and cover roadwarrior config as an afterthought.

My weapon of choice distro-wise is CentOS, however these instructions could be applied to any other distro (ie, Ubuntu) with a basic understanding of your particular platform's networking configuration methods. Really, OpenVPN can run on just about anything, including Windows, if you feel so inclined. However, you'll probably get the most bang for your buck (free) using Linux. In my test environment, running stock CentOS 6.4, I had the scenario below running on a server with less than 90MB of total system memory usage.

Why CentOS?
(BEGIN RANT)
I have two main reasons for promoting the use of CentOS:
1) CentOS is based upon RedHat Enterprise Linux (RHEL), which currently enjoys status as the most vendor-supported OS in enterprise environments. Translation? Knowledge of an RHEL derivitaves like CentOS is marketable skill to put on your resume. Yes, Ubuntu is gaining popularity in tech circles, but still doesn't compare to RHEL for vendor support. There's a reason both VMWare and Citrix use RHEL derivitaves as their baremetal OS.
2) CentOS has long-term-support (LTS). I've used Fedora for years, and I enjoy(ed) playing with some of the bleeding-edge features it offers. But the bleeding edge is on a double-edged sword. Fedora has a relatively aggressive release and support schedule. Install Fedora X, and expect that Fedora Z will replace it in about a year, leaving version X effectively without support. This gets to be a pain when you need to "yum update" your system just a year or so after you installed it. CentOS however has a support schedule that will ensure you likely have updates for far more years than the lifecycle of your hardware. For example CentOS 6, released in 2011, reaches EOL in late 2020. That's almost 10 years of support, on a free platform!
(END RANT)

First, let's diagram the network we are going to design:

Remote Office 1 (10.101.0.x/24)
|
VPN Tunnel (10.200.101.1/30)
|
Main Office (10.100.0.x/24)
|
VPN Tunnel (10.200.102.1/30)
|
Remote Office 2 (10.102.0.x/24)

For this guide, we're going to assume you want all remote offices to have routing enabled to each other (via the Main office).

Note that the IP addresses above are just for example. You could create your own IP addressing scheme with FAR better utilization of the the private address space. The subnets I have used (ie 10.200.x.x/16) are just for increased clarity.

Each OpenVPN server/endpoint can have one interface (assuming you are doing NAT/firewalling elsewhere on your network), OR you could have dual interfaces: One on the LAN, the other on the WAN/Internet connection. It's up to you depending on where you want to do your firewalling. For the purposes of this guide, we're putting the VPN server behind another firewall.

On your firewall
Forward UDP port 1194 for your external internet connection to the internal IP of your VPN servers/endpoints, ie:

Main Office 10.100.0.5:1194
Remote Office 1 10.101.0.5:1194
Remote Office 2 10.102.0.5:1194

If for some reason UDP/1194 were blocked by your ISP, you could switch to something like TCP/80, but for the purposes of this guide, we'll stick with the default UDP/1194.

On each server/endpoint
Install your base OS. I chose to just install CentOS 6.4 minimal installation with the default options. Once installed, get networking set up and run

yum update

and get everything up-to-date.

Install the EPEL Repo configs

yum install http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

Turn off the built-in firewall (remember, we're using a 3rd party device for this). You could leave it on, but then you'd have to configure it to pass traffic to/from your VPN tunnel interface. I'll leave that up to you to figure out should you choose to do so.

chkconfig iptables off ; service iptables stop

Install the OpenVPN packages and dependencies:

yum install openvpn

Server (Main Office)

Copy the the easy-rsa environment to /etc/openvpn/easy-rsa, do some config prep, and alter the vars file

cp -rp /usr/share/openvpn/easy-rsa/2.0 /etc/openvpn/easy-rsa
cd /etc/openvpn/easy-rsa/
cp -rp vars vars.orig
cp -rp openssl-1.0.0.cnf openssl.cnf
vi vars

Go down to "KEY_COUNTRY" and edit the Country, Province, etc, down to the OU, ie:

export KEY_COUNTRY="US"
export KEY_PROVINCE="MN"
export KEY_CITY="Minneapolis"
export KEY_ORG="Muchtall"
export KEY_EMAIL="me@my.domain.com"
export KEY_EMAIL=me@my.domain.com
export KEY_CN=vpn.mydomain.com
export KEY_NAME=MuchtallOpenVPNServer
export KEY_OU=Muchtall

Now generate your server's certificate authority:

. ./vars
./clean-all
./build-ca

Accept the defaults for the prompts (we already set them)

Now build the server key

./build-key-server vpn.mydomain.com

Similar to above, accept the default prompts. You will have to answer "y" to the questions "Sign the certificate? [y/n]:" and "1 out of 1 certificate requests certified, commit? [y/n]"

Now generate certs/keys for each remote site

./build-key remote-office-1
./build-key remote-office-2

Rinse, repeat on the prompts.

And generate the Diffie-Hellman parameters:

./build-dh

Great! Our certificates are all set up!

Next time you want to generate a new client key, just run

cd /etc/openvpn/easy-rsa/
. ./vars
./build-key remote-office-3

Now, let's set up the configs. There's a sample config at /usr/share/doc/openvpn-2.2.2/sample-config-files/server.conf, however we're going to set one up using this template, just to keep things simple:

port 1194
proto udp
dev tun
ca easy-rsa/keys/ca.crt
cert easy-rsa/keys/vpn.mydomain.com.crt
key easy-rsa/keys/vpn.mydomain.com.key
dh easy-rsa/keys/dh1024.pem
server 10.200.0.0 255.255.0.0
ifconfig-pool-persist ipp.txt
# Tell clients that we can handle routes for these networks
push "route 10.100.0.0 255.255.0.0"
push "route 10.101.0.0 255.255.0.0"
push "route 10.102.0.0 255.255.0.0"
client-config-dir ccd
# Tell OpenVPN that it routes for anything within these subnets
route 10.101.0.0 255.255.0.0
route 10.102.0.0 255.255.0.0
client-to-client
keepalive 10 120
comp-lzo
persist-key
persist-tun
status openvpn-status.log
verb 3

Now create the ccd directory

mkdir /etc/openvpn/ccd

And lets create the client-specific configs to route each individual subnet to the respective site:

# In /etc/openvpn/ccd/remote-office-1
iroute 10.101.0.0 255.255.255.0

And

# In /etc/openvpn/ccd/remote-office-2
iroute 10.102.0.0 255.255.255.0

Good? Good!

Normally, IP subnets for the tunnels are allocated as new tunnels connect to the server. Let's pre-set the IPs for each tunnel. This part isn't necessary, however I like to do this to assist with clarity in troubleshooting with traceroutes. In /etc/openvpn/ipp.txt:

remote-office-1,10.200.101.4
remote-office-2,10.200.102.4

And re-set the SELinux permissions on the ipp.txt file

restorecon -v './ipp.txt'

Now we're ready to start the the OpenVPN service up:

service openvpn restart

Check the syslog to see if anything serious got spit out:

tail -100 /var/log/messages

You should see something similar to this:

May 7 14:54:53 mainoffice openvpn[13362]: OpenVPN 2.2.2 x86_64-redhat-linux-gnu [SSL] [LZO2] [EPOLL] [PKCS11] [eurephia] built on Aug 10 2012
May 7 14:54:53 mainoffice openvpn[13362]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
May 7 14:54:53 mainoffice openvpn[13362]: Diffie-Hellman initialized with 1024 bit key
May 7 14:54:53 mainoffice openvpn[13362]: TLS-Auth MTU parms [ L:1542 D:138 EF:38 EB:0 ET:0 EL:0 ]
May 7 14:54:53 mainoffice openvpn[13362]: Socket Buffers: R=[229376->131072] S=[229376->131072]
May 7 14:54:53 mainoffice openvpn[13362]: ROUTE default_gateway=10.0.3.2
May 7 14:54:53 mainoffice openvpn[13362]: TUN/TAP device tun0 opened
May 7 14:54:53 mainoffice openvpn[13362]: TUN/TAP TX queue length set to 100
May 7 14:54:53 mainoffice openvpn[13362]: /sbin/ip link set dev tun0 up mtu 1500
May 7 14:54:53 mainoffice openvpn[13362]: /sbin/ip addr add dev tun0 local 10.200.0.1 peer 10.200.0.2
May 7 14:54:53 mainoffice openvpn[13362]: /sbin/ip route add 10.101.0.0/16 via 10.200.0.2
May 7 14:54:53 mainoffice openvpn[13362]: /sbin/ip route add 10.102.0.0/16 via 10.200.0.2
May 7 14:54:53 mainoffice openvpn[13362]: /sbin/ip route add 10.200.0.0/16 via 10.200.0.2
May 7 14:54:53 mainoffice openvpn[13362]: Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
May 7 14:54:53 mainoffice openvpn[13369]: UDPv4 link local (bound): [undef]:1194
May 7 14:54:53 mainoffice openvpn[13369]: UDPv4 link remote: [undef]
May 7 14:54:53 mainoffice openvpn[13369]: MULTI: multi_init called, r=256 v=256
May 7 14:54:53 mainoffice openvpn[13369]: IFCONFIG POOL: base=10.200.0.4 size=16382
May 7 14:54:53 mainoffice openvpn[13369]: IFCONFIG POOL LIST
May 7 14:54:53 mainoffice openvpn[13369]: remote-office-1,10.200.101.4
May 7 14:54:53 mainoffice openvpn[13369]: remote-office-2,10.200.102.4
May 7 14:54:53 mainoffice openvpn[13369]: Initialization Sequence Completed
May 7 14:54:53 mainoffice kernel: tun0: Disabled Privacy Extensions

Once you've verified based upon the above output that everything is running fine, go ahead and mark the service to start automatically

chkconfig openvpn on

Client Configs (Remote Offices)
On each server, create the file "/etc/openvpn/vpn.mydomain.com.conf", and populate with the following:

client
dev tun
proto udp
remote 192.168.56.100 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
cert remote-office-1.crt
key remote-office-1.key
comp-lzo
verb 3

Be sure to change the cert name as appropriate.

Copy the ca.crt, remote-office-1.key, and remote-office-1.crt to the /etc/openvpn/ directory of the client. Repeat for Office 2.

Set the permissions on the key file so that it can't be copied by non-root users.

chmod 600 /etc/openvpn/remote-office-1.key

Start the OpenVPN service

service openvpn start

Check the output of syslog for similar output:

May 7 16:26:39 remote-office-1 openvpn[1566]: OpenVPN 2.2.2 x86_64-redhat-linux-gnu [SSL] [LZO2] [EPOLL] [PKCS11] [eurephia] built on Aug 10 2012
May 7 16:26:39 remote-office-1 openvpn[1566]: WARNING: No server certificate verification method has been enabled. See http://openvpn.net/howto.html#mitm for more info.
May 7 16:26:39 remote-office-1 openvpn[1566]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
May 7 16:26:39 remote-office-1 openvpn[1566]: LZO compression initialized
May 7 16:26:39 remote-office-1 openvpn[1566]: Control Channel MTU parms [ L:1542 D:138 EF:38 EB:0 ET:0 EL:0 ]
May 7 16:26:39 remote-office-1 openvpn[1566]: Socket Buffers: R=[229376->131072] S=[229376->131072]
May 7 16:26:39 remote-office-1 openvpn[1566]: Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
May 7 16:26:39 remote-office-1 openvpn[1566]: Local Options hash (VER=V4): '41690919'
May 7 16:26:39 remote-office-1 openvpn[1566]: Expected Remote Options hash (VER=V4): '530fdded'
May 7 16:26:39 remote-office-1 openvpn[1567]: UDPv4 link local: [undef]
May 7 16:26:39 remote-office-1 openvpn[1567]: UDPv4 link remote: x.x.x.x:1194
May 7 16:26:39 remote-office-1 openvpn[1567]: TLS: Initial packet from x.x.x.x:1194, sid=f439995e ac9dd302
May 7 16:26:39 remote-office-1 openvpn[1567]: VERIFY OK: depth=1, /C=US/ST=MN/L=Minneapolis/O=Muchtall/OU=Muchtall/CN=vpn.mydomain.com/name=MyOpenVPNServer/emailAddress=me@my.domain.com
May 7 16:26:39 remote-office-1 openvpn[1567]: VERIFY OK: depth=0, /C=US/ST=MN/L=Minneapolis/O=Muchtall/OU=Muchtall/CN=vpn.mydomain.com/name=MyOpenVPNServer/emailAddress=me@my.domain.com
May 7 16:26:39 remote-office-1 openvpn[1567]: Data Channel Encrypt: Cipher 'BF-CBC' initialized with 128 bit key
May 7 16:26:39 remote-office-1 openvpn[1567]: Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
May 7 16:26:39 remote-office-1 openvpn[1567]: Data Channel Decrypt: Cipher 'BF-CBC' initialized with 128 bit key
May 7 16:26:39 remote-office-1 openvpn[1567]: Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
May 7 16:26:39 remote-office-1 openvpn[1567]: Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 1024 bit RSA
May 7 16:26:39 remote-office-1 openvpn[1567]: [vpn.mydomain.com] Peer Connection Initiated with 192.168.56.150:1194
May 7 16:26:41 remote-office-1 openvpn[1567]: SENT CONTROL [vpn.mydomain.com]: 'PUSH_REQUEST' (status=1)
May 7 16:26:41 remote-office-1 openvpn[1567]: PUSH: Received control message: 'PUSH_REPLY,route 10.100.0.0 255.255.0.0,route 10.101.0.0 255.255.0.0,route 10.102.0.0 255.255.0.0,route 10.200.0.0 255.255.0.0,topology net30,ping 10,ping-restart 120,ifconfig 10.200.101.6 10.200.101.5'
May 7 16:26:41 remote-office-1 openvpn[1567]: OPTIONS IMPORT: timers and/or timeouts modified
May 7 16:26:41 remote-office-1 openvpn[1567]: OPTIONS IMPORT: --ifconfig/up options modified
May 7 16:26:41 remote-office-1 openvpn[1567]: OPTIONS IMPORT: route options modified
May 7 16:26:41 remote-office-1 openvpn[1567]: ROUTE default_gateway=10.0.3.2
May 7 16:26:41 remote-office-1 openvpn[1567]: TUN/TAP device tun0 opened
May 7 16:26:41 remote-office-1 openvpn[1567]: TUN/TAP TX queue length set to 100
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip link set dev tun0 up mtu 1500
May 7 16:26:41 remote-office-1 kernel: tun0: Disabled Privacy Extensions
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip addr add dev tun0 local 10.200.101.6 peer 10.200.101.5
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip route add 10.100.0.0/16 via 10.200.101.5
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip route add 10.101.0.0/16 via 10.200.101.5
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip route add 10.102.0.0/16 via 10.200.101.5
May 7 16:26:41 remote-office-1 openvpn[1567]: /sbin/ip route add 10.200.0.0/16 via 10.200.101.5
May 7 16:26:41 remote-office-1 openvpn[1567]: Initialization Sequence Completed

And verify that the routing is taking place

ping 10.100.0.1

Set the openvpn service to start automatically

chkconfig openvpn on

Repeat these steps for the Office 2 client. And verify that you can ping across both tunnels to Office 1

Roadwarrior Config (this is optional)
As an afterthought, I said I'd cover Roadwarrior configuration. Here's a basic rundown:

- Generate a new cert/key pair for your username using the above ./build-key commands
- Install OpenVPN for Windows (if you're on Mac or Linux, you likely know how to do this already)
- Copy the ca.crt, username.crt, and username.key files to "C:\Program Files\OpenVPN\config\"
- Create a config file named C:\Program Files\OpenVPN\config\vpn.mydomain.com.ovpn with these contents:

client
dev tun
proto udp
remote vpn.mydomain.com 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
cert myusername.crt
key myusername.key
comp-lzo
verb 3

Right-click on the OpenVPN GUI in the taskbar and click "Connect".

Quick Tip: Identifying Space Consumption in Linux via Command line

Firstly, let me strongly recommend JDiskReport if a GUI is available to you. It's super easy to use and helps you quickly drill down into the disk and identify disk space usage, on any platform with Java support.

In lieu of that, if you have need to identify disk usage via the command line, run this set of commands:
find / -exec du -ks {} \; 2> /dev/null | sort -n | awk '{printf $1 "\t"; if (system("test -d \""$2"\"")) { print $2 } else { print $2 "/" } }' | tail -1000

This will spit out a list of the largest 1000 individual files and folder sums, sorted by size.

Depending on the size of the disk, this will take a while, as should be expected as it runs a "du" for each path it finds. Caching helps if/when you re-run this.

Fixing Xen Error: (12, 'Cannot allocate memory')

I don't know why this hasn't already been posted all over the interwebs, so I thought I'd post it here. It's amazing how some of the simplest answers never seem to make it into prominence in the Google search rankings.

Anyway, after cloning a VM today in Xen, I tried to start the DomU, and got this error:

Error: (12, 'Cannot allocate memory')

I assumed it had something to do with my Dom0 memory cache consuming nearly every last bit of free memory on my system, so I ran this to clear the cache:

echo 3 > /proc/sys/vm/drop_caches

I confirmed that that "free" reported that the cached memory dropped near zero. Great! Except it didn't help. I then ran "xm info" to check the memory usage in Xen

total_memory : 16378
free_memory : 2

Odd. I checked the Dom0 status to see what's up, and sure enough, my memory allocation on Dom0 is way high.

xm list Domain-0
Name ID Mem VCPUs State Time(s)
Domain-0 0 10482 4 r----- 3273110.4

But I still didn't know how to shrink it. I knew it had something to do with memory ballooning. I also knew you could set the memory allocation of a DomU with "xm mem-set", but I didn't know anything about how to do it with the Dom0. A quick IRC message to my Xen guru, Jima, and I get this simple command (obvious really) in response:

xm mem-set Domain-0 512

This balloons the Dom0 host system (as well as any domain, really) memory usage down to 512MB. In my case, my Dom0 claimed to be using about 2 Gig already, so I ballooned down to 4G instead. Not sure if that was just due to some bad math or assumptions on my part, as I never run anything of significance in Dom0. After you run this, you should be able to start your DomU just fine.

Installing PCL6 Drivers to a Samba/CUPS Print Server

First, a little background...

The classic procedure for installing drivers to a printer shared via CUPS with a raw Samba frontend, usually is roughly something like this:

  1. Create printer in CUPS
  2. Open shared printer properties in Windows
  3. Install drivers via Advanced tab

For me, this worked great, most of the time, assuming that the driver was either PS or PCL5(e). However, PCL6 drivers rarely loaded properly. They crashed upon upload, and the driver rarely ever worked, if point 'n print even installed the driver. As I understand it, the root of the problem is that the drivers need a valid "form" database specific to the driver to be stored on the server in order to load, but valid form data can only be created by executing the driver. On Windows servers, this isn't a problem, but Samba servers cannot execute the windows driver code in order to generate this form. PCL6 drivers are particularly finicky about this, and often times will refuse properly copy to the server.

The solution I've come up with is to load the driver entirely on a Windows PC first, and tell Samba to copy the drivers and related forms, ACLs, etc over. Some of the values I use here for the printer names, etc, are very generic and only specified to match the naming standards in my environment. Feel free to change your printer setup to suit your needs.

The Procedure
On a Windows PC:

  1. Install the printer as you would normally for TCP/IP printing on a windows PC.
  2. Rename the printer to the share name you will be using (i.e., CopierCP5000)
  3. Open the printer properties
  4. Share the printer using the same name (CopierCP5000)
  5. Set the printer description to the printer name (i.e., Copier CP 5000)
  6. Set the printer location to the office name and city (ACME Abu Dhabi)
  7. Usually under the Accessories/Options tab, update/get information from the printer
         This ensures that we obtain all the correct hardware configuration information for the device (duplexer, trays, hole punch, stapler, etc)
  8. Open the Advanced tab and click the "Printing Defaults" button
  9. Click on each tab and allow the tab to load
         This ensures that the form values for each tab are created.
  10. Still inside the Printing Defaults, change the page orientation to Landscape, hit apply, change the page orientation to Portrait, and hit apply. Click OK.
  11. Close the printer properties by clicking OK

Next, on the server:

### Set up the printer in CUPS, i.e.:
lpadmin -p CopierCP5000 -L "ACME Abu Dhabi" -D "Copier CP 5000" -v lpd://192.168.xx.xx -m raw -E
smbcontrol smbd reload-config
 
### Copy the drivers, forms, ACLs, etc from the Windows PC
net rpc printer MIGRATE ALL CopierCP5000 -S mypcname -Uusername

Finally, test the driver in Windows:

  1. Open the local Printers control panel (shortcut command below for Win7 users)
         rundll32 shell32.dll,SHHelpShortcuts_RunDLL PrintersFolder
  2. Delete your shared printer
  3. Right-click in the control panel window and select "Server Properties"
  4. Open the Drivers tab
  5. Select your driver and click "Remove".
  6. Proceed with removal and close the server properties window.
  7. Open \\servername\
  8. Double-click your printer
  9. Print a test page
         If this fails, try restarting the local PC's Print Spooler service

So far I've only tested this on a few of our known-problematic printers that have PCL6 drivers available, and it's worked perfectly in each case. However it you notice this doesn't work for you, post a comment and I'll see what I can do to help.

Restreaming ASF to MP3 Via Icecast and VLC

Say you have an internet radio stream that you like to listen to. You like to actually get up from your computer once in a while and go outside, so you want to play it on your iPhone/Blackberry/G1/Treo/Centro/Pre/Whatever as you're out and about. But wait! Due to poor decision making by your beloved stream provider, your favorite stream is only in Microsoft's poorly-supported ASF streaming format. There's no MP3 or OGG options! Many if not all of the phones I just mentioned may/will not play these streams, at least not natively. MP3 on the other hand, is ubiquitous (though, given, not truly as free as OGG). One option you may have, if you run your own Linux server, is to download the ASF stream, re-encode it to MP3, and re-stream it via your own Icecast server. Icecast is easy enough to set up, however information on setting up the VLC portion of this workaround online seems a little slim online, so I thought I'd make a post simplifying the setup.

First off, check to make sure your VLC has "Shout" Shoutcast/Icecast output support:

vlc --list | grep access_output_shout

You should see the following in the output:

access_output_shout IceCAST output

If not, you may need to either re-install with a version of VLC that supports shout output, or recompile it with shout support.

Then run the following command, altering the items in bold for your purposes (all on one line):

cvlc "http://classicalstream2.publicradio.org/classical" ':sout=#transcode{acodec=mp3,ab=192,channels=2}:duplicate{dst=std{access=shout,mux=raw,dst=source:YOURPASSWORD@your.icecast.server.com:8000/mpr.mp3}}' --sout-shout-mp3 --sout-shout-name="MPR MP3 Restream" --sout-shout-description="Restreaming of MPR" --sout-shout-url="http://minnesota.publicradio.org" --sout-shout-genre="Liberal Tripe" --loop

Assuming you have already properly set up your Icecast server, you should now have a mountpoint running the stream you have just created.

Note that VLC seems to have problems parsing some ASX playlist files. You may need to download the ASX file with wget to investigate what the actual stream URL is. For example, the stream above came from "http://minnesota.publicradio.org/tools/play/streams/classical.asx", which had these contents:

<ASX version="3.0">
   <ABSTRACT>The Windows Media Stream for MPR's Classical Music Service</ABSTRACT>
   <TITLE>MPR Classical Windows Stream</TITLE>
   <AUTHOR>Minnesota Public Radio</AUTHOR>
   <COPYRIGHT>2009 Minnesota Public Radio</COPYRIGHT>
   <MOREINFO HREF = "http://minnesota.publicradio.org" />
   <ENTRY>
      <TITLE>MPR Classical Windows Stream Underwriter</TITLE>
      <COPYRIGHT>2009 Minnesota Public Radio</COPYRIGHT>
      <REF HREF="http://ondemand2.publicradio.org/ondemand/joinnow.wma"/>
      <BANNER HREF="http://www.publicradio.org/applications/underwriters/images/prepend/content/joinnow.gif">
          <MOREINFO HREF = "https://contribute.publicradio.org/"></MOREINFO>
      </BANNER>
   </ENTRY>
   <ENTRY>
      <TITLE>MPR Classical Windows Stream</TITLE>
      <COPYRIGHT>2009 Minnesota Public Radio</COPYRIGHT>
      <REF HREF="http://classicalstream2.publicradio.org/classical"/>
      <BANNER HREF="http://www.publicradio.org/applications/underwriters/images/prepend/content/joinnow.gif">
          <MOREINFO HREF = "https://contribute.publicradio.org/"></MOREINFO>
      </BANNER>
   </ENTRY>
</ASX>

As you can see, the first item in the playlist is NPR's obligatory call (**cough** commercial **cough**) for donations (http://ondemand2.publicradio.org/ondemand/joinnow.wma). The second is your actual stream URL (http://classicalstream2.publicradio.org/classical). You'll want to use the stream URL in your VLC command.

Also, I have seen cases where VLC pukes on some ASF streams. Though I haven't confirmed this, it appears to be a bug with downsampling/resampling (http://mailman.videolan.org/pipermail/vlc-devel/2009-February/056823.html). In these cases, you may need to look that the 1.0.0 prerelease versions, which may not be available on your distro release (Fedora, for example, doesn't have it until 11). Good luck!

UPDATE 10/27/09: It would appear that the precompiled RPMs offered on the VLC site for 1.0 or higher do not contain support for shout output. Not a big deal. Just download the source RPM (e.g., from http://download1.rpmfusion.org/free/fedora/development/source/SRPMS/) install the sources (rpm -Uvh vlc-1.0.3-0.1_rc.fc12.src.rpm), and edit the vlc.spec file (usually found in ~/rpmbuild/SPECS/). Look for a line that says "--enable-realrtsp \" and add a line below it that says "--enable-shout \". Save and close the file, then run "rpmbuild -ba vlc.spec". After it recompiles, install the updated RPMs (cd ~/rpmbuild/RPMS/x86_64/ ; yum --nogpgcheck reinstall vlc-core-1.0.3-0.1_rc.fc11.x86_64.rpm vlc-1.0.3-0.1_rc.fc11.x86_64.rpm). Make sure to alter the above paths to reflect your architecture, Fedora release, and vlc version.

Deleting Specific Entries from Conntrack in Linux

First, use conntrack to correctly identify your entries:

    conntrack -L -s 172.16.1.45 -d 123.123.123.123

This should display any connections that came from the internal IP of 172.16.1.45 destined to 123.123.123.123

Once you have confirmed the connections shown are the ones you with to delete/reset, paste the following after the command from above:

    conntrack -L -s 172.16.1.45 -d 123.123.123.123 | sed 's/=/ /g' | awk '{print("conntrack -D -s "$6" -d "$8" -p "$1" --sport="$10" --dport="$12)}'

This will print a list of the commands that would run to delete the connections. Replace "print" with "system" to execute the deletions:

    conntrack -L -s 172.16.1.45 -d 123.123.123.123 | sed 's/=/ /g' | awk '{system("conntrack -D -s "$6" -d "$8" -p "$1" --sport="$10" --dport="$12)}'

Finally, re-run the list command to see that all the entries have been removed:

    conntrack -L -s 172.16.1.45 -d 123.123.123.123

And you're done!

Changing /dev permissions on Fedora

It's infrequent enough that I have to do this that I forget how. So to help me remember, I'm posting it here:

To make your permissions changes to devices in /dev/ survive reboots and become permanent, you need to alter the device permissions in

    /etc/security/console.perms.d/50-default.perms

So for example, say I have a created a group that holds all users on the system that are to have RW access to the video capture hardware (such as mythtv and zoneminder), open the perms file and look for this line:

    0600 0600 root

This v4l class covers all /dev/video* devices. To make them RW by the group "video", change the line to look like this:

    0660 0660 root.video

You can reboot to see the effect.

A Simple Uptime Output Method

If you've ever tried to parse the output of "uptime", you know it can be complicated. There's a simpler solution: Use the raw numbers from /proc/uptime. Here's a simple script that spits out the days and remaining time in digital clock format:

    cat /proc/uptime | awk '{printf "%d%s%.2d%s%.2d%s%.2d%s" , $1/86400 , " days, " , $1%86400/3600 , ":" , $1%86400%3600/60 , ":" , $1%86400%3600%60 , "\n"}'

Note that the output can be changed to your needs. See the AWK Tutorial for more info on using modulus (%) in calculations, and string formatting (i.e., %d and %s)

Recovering from a dd_rhelp add_chunk error

If you've got a crashed disk, dd_rhelp is a very
useful tool for quickly recovering a large amount of data from just
about any failed disk. I used it earlier this week to help recover a
160GB linux LVM group member drive. Unfortunately, after a while of
running, it hit this error:

    dd_rhelp: error: add_chunk : invalid argument '0-(info)' (is not correctly formatted as number:number)

Here's
how you get past that. The problem is that dd_rhelp was not able to
determine the size of your hard disk automatically, so it is scanning
beyond the actual size, which of course fails. We need to tell it our
disk size manually.

First, make a backup of your logfile, for example:
    cp -rp sdb.log sdb.log.bad-eof
That way you can start over if you mess up.

Go to the first line in the original log file that states:
    dd_rescue: (warning): /dev/sda (282662620.0k): No space left on device!
Or something similar. Scroll up to the next line before that that reads:
    === COMPUTED VERSION OF LOG :
Delete all lines including and following this line, then save your file.

Obtain your filesystem size with fdisk:
    # fdisk -l /dev/sda
   
    Disk /dev/sda: 160.0 GB, 160000000000 bytes
    and so on...

In this case, your filesystem size would be 160000000000 bytes, or 160000000  kbytes.

Next,
change all the lines in the file stating "eof:nothing" to
"eof:160000000". Use your fs size in replacement of 160000000 kbytes.
You can quickly do this using sed:
    sed 's/eof:nothing/eof:160000000/g' -i sdb.log

Now re-run your dd_rhelp command. It should now continue to recover as normal!

Thanks to Valentin from dd_rhelp and "Master One" for leading me in the right direction.

UPDATE:
It seems that after a short time, I ran into the same error again. My
solution was to open the dd_rhelp script, look for the line that says:
    eof="nothing"
and change it to
    eof="160000000"
(of course, substituting your disk size in kb)

Re-do the cleanup above and re-run.

UPDATE 2: So I've found that dd_rhelp is slow, or at least, it's slow on my damaged disk. I suggest trying ddrescue instead (not to be confused with dd_rescue from above). It seems to be much more efficient at sidestepping bad data and gives a more concicse summary of recovery progress. This includes how much data has been recovered so far, lost so far, number of total errors, and transfer rates. For more info on the design differences between ddrescue and dd_rhelp, see the SpinRite wikipedia entry.

Creating a Hylafax Dropbox (No Email)

The default setup for Hylafax is to drop faxes into an email and send them off to the "FaxMaster". This weekend I needed to stop hylafax from doing that, and instead just drop it into a folder. Best I can tell after a few hours of searching Google and perusing code, hylafax on it's own does not do this. There appears to be no options for it in faxrcvd.

So instead of searching any longer, I decided to write my own, very simple, yet perfectly funtional faxrcvd script. All it does is take the received TIFF image and converts it to PDF (the preference of my users), and drops it into a specified folder. Here's the code, feel free to use it:

# Make a backup of the original fax received script (faxrcvd emails faxes).
cp -p /var/spool/hylafax/bin/faxrcvd /var/spool/hylafax/bin/faxrcvd.orig

EDIT: I received a message the other day saying that my script didn't work. It's possible some of the code got messed up by Nuke, so I'm just suggesting downloading instead...

# Download the MuchTall faxrcvd code in replacement of the existing code
wget -O - /"content/faxrcvd > /var/spool/hylafax/bin/faxrcvd

# Edit it with your favorite editor to change the path and permissions
vi /var/spool/hylafax/bin/faxrcvd
    OR
nano -N /var/spool/hylafax/bin/faxrcvd

Change the path and permissions (chmod) to what you want. In my case, I needed to allow all my users R/W access to these faxes.

There you go! Assuming you already have configured hylafax, you should be all set!

Cleaning up A**hole Spam in PHPNuke

When setting up PHPNuke, you really have to remember to turn on captcha checking for registrations first thing in config.php (gfx_chk). If you don't, you'll get pummeled with comment spam in a matter of hours.

If, like me, you messed up and forgot to, you'll have to delete those bogus accounts, and delete the comments. Deleting the comments can be tedious. So instead, run a command from a unix shell to generate some html content that will make your life easier...

mysql -u dbusername -pdbpassword dbname -e 'SELECT tid,sid FROM nuke_comments;' | grep -v tid | awk '{print "<a href=\"http://yourdomain.com/admin.php?op=RemoveComment&tid=" $1 "&sid=" $2 "&ok=1\">http://yourdomain.com/admin.php?op=RemoveComment&tid=" $1 "&sid=" $2 "&ok=1</a>"}'

This will output a bunch of HTML that you can insert into a page. Click on each of those links to delete a comment. Much quicker than searching for them all.

Now, I suppose someone could write a PHP module for Nuke that would automate this. I'm just too lazy to ;-)

Setting up Tiny MCE 2.1.1 with Advanced Editor on PHP-Nuke 7.9

I just got Tiny MCE updated on MuchTall and I just thought I'd pass the update instructions on

1) Download and unpack Tiny MCE from http://tinymce.moxiecode.com/download.php
2) Move the folder /includes/tiny_mce to /includes/tiny_mce.orig
3) Copy the new tinymce/jscripts/tiny_mce folder to /includes/tiny_mce
4) Download this javascript.php.patch patch into /includes/
5) Run "patch -p0 < javascript.php.patch" from within /includes/
6) Open your PHPNuke editor page

That's it! Now you have a rich text editor with advanced editing features right within PHPNuke!

Windows is Bad for Bind Zone Files

A buddy of mine was recently setting up bind on his linux box and had an issue. His zone files wouldn't load without spewing out errors:

Jun 8 10:33:05 server named: hisdomain.com:18: unknown RR type 'host1'
Jun 8 10:33:05 server named: hisdomain.com:19: unknown RR type 'host2'
Jun 8 10:33:05 server named: hisdomain.com:20: unknown RR type 'host3'
Jun 8 10:33:05 server named: hisdomain.com:23: unknown RR type 'host4.hisdomain.com.'
Jun 8 10:33:05 server named: dns_master_load: hisdomain.com:24: isc_lex_gettoken() failed: unexpected end of input
Jun 8 10:33:05 server named: dns_master_load: hisdomain.com:24: unexpected end of input
Jun 8 10:33:05 server named: hisdomain.com: file does not end with newline
Jun 8 10:33:05 server named: zone hisdomain.com/IN: loading from master file hisdomain.com failed: unknown class/type
Jun 8 10:33:05 server named: _default/hisdomain.com/in: unknown class/type

Fun stuff. After messing around with his domain files for about an hour, I realized something. There were all these excessive spaces after each line. So I opened vi and proceeded to remove them all, as well as replacing spaces in the doc where a tab should be. Lo and behold, it loaded fine.

It appears the the root of the problem were those "spaces" at the end of the line. I figure they were probably DOS carriage returns created my some lame windows text editor. Once removed, the file loads fine.

Move Your Firefox Close Tab Button

I recently had to re-install my PC and I decided to take the opportunity to freshen up my Firefox profile by abandoning my prefs.js file and only migrating settings and extensions as needed.

As part of this, I ran into an issue I dealt with a while back and had to look up once more.

http://ffextensionguru.wordpress.com/2006/10/22/firefox-20-close-tab-buttons/

I don't like having my little red X to close the tabs actually in the tab. I like it at the end of the tab-bar. Just a quick rundown of how to change this:

Browse to about:config and filter for browser.tabs.closeButtons. Set that value to 3.

Change Your Default syslog Options to Disable DNS Lookups

Disclaimer: Nevermind. Turns out that this only disables lookups on remote syslog entries (from another syslog host). Still looking for a solution on this one. Feel free to read on if you wish.

So this weekend, someone (as happens quite frequently) attempted to gain access to my server via the FTP daemon. No big surprise there. What was interesting about it was that there was one attempt from an rhost named "oa". Not oa.1337hacker.com, or even an IP, just.... oa. As expected an nslookup of "oa" yielded nothing.

So I ended up turning off reverse lookups on syslog. I'd suggest you do the same. You can't really trust reverse lookups. What's to prevent someone who has control over their PTR record to create any bogus reverse lookup, even google.com or yourdomain.com and attempt the same thing? You have no means at that point to actually identify the attacker positively by IP.

To turn off reverse lookups for syslog on FC6, just add -x to this line in /etc/sysconfig/syslog:

SYSLOGD_OPTIONS="-x ... "

See here for my bug report on Redhat's Bugzilla. https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=227357

MuchTallWare: ftpautoban.pl: Auto-ban IPs from VSFTP

Some dumbass has been pummeling my ftp server with brute-force FTP attempts. So I've written a script that you can add to your cron tab to help you auto-ban them via hosts.deny. Pay attention to the variables as this was written on a RedHat 9 box and may not be entirely applicable to other platforms or versions.

ftpautoban.pl

Adding Firewall rules on DD-WRT

Just recently I got OpenVPN set up on my WRT54G (w/DD-WRT) to connect to the network at my workplace. However, I noticed that although the vpn tunnel was up, packets weren't traversing it. For a while I thought it was some odd routing issue, but then I realized that the issue was with the firewall config. I just assumed that the firewall config was automatically changed to allow VPN tunnel traffic.

So here's the problem. DD-WRT is meant, on it's face, to be a cutesy interface for average Joes, with some power-user features. I doesn't allow you to set custom firewall rules via the web interface. Furthermore, there's no flat file you can edit within the console to make changes. All configs get regenerated and overwritten on boot. The configuration alterations are actually stored in nvram. To see this data, just ssh to your DD-WRT and enter:

nvram show

Ahh. Brings back memories of configuring SpeedStream routers from scratch. Anyhow... Here's where you can see all the config that the DD-WRT uses to actually generate the volatile standardized conf files that sit in the ramdisk. You can actually see individual configuration variables if you'd like to alter or add to them by hand (instead of by web). For example, this will show you the NAT forwarding config:

nvram get forward_spec

You should see a list of strings showing the NAT config. It's all on one line with options separated by colons (and a ">") and delimited by spaces. That is, assuming you have any NAT entries configured.

But here's the fun part. I have 2 iptables rules that need to be inserted when the system boots to allow the VPN tunnel's traffic to pass:

iptables -I INPUT -i tun+ -j ACCEPT
iptables -I FORWARD -i tun+ -j ACCEPT

First, check to make sure you don't have any existing config that you may have to add:

nvram get rc_firewall

I can add this to the startup config by sending following commands:

nvram set rc_firewall="iptables -I INPUT -i tun+ -j ACCEPT
iptables -I FORWARD -i tun+ -j ACCEPT"
nvram commit

Copy this into notepad, edit it as you see fit, and paste it into your SSH session. After running them, just reboot to apply the change.

UPDATE 11/17/06: Apparently my habit of making solutions geekier than they have to be has bitten me again. It appears that if you simply browse to Administration > Commands (http://yourrouter/Diagnostics.asp), you can enter the commands there and hit "Save Firewall" to achieve the same effect. Damn. And here I thought I discovered something cool :-)

Adding RSS and Atom Feeds to PHP-Nuke

I recently updated Muchtall.com to serve RSS and Atom newsfeeds for the site. For more info on how to do this to your own PHP-Nuke site, click here:

RSS and Atom for PHP-Nuke

Spybot Search and Destroy AutoUpdate / AutoImmunize

It's not a linux tip, but it's still a tip...

If you're looking for an easy way to keep Spybot S&D updated and your system immunized, try this registry file which launches Spyboy S&D, updates, and immunizes in the background on logon.

Spybot AutoUpdate and AutoImmunize On Logon.reg

How IPCop Rocked My World

A few weeks ago at my job, we started up a new mini-branch in an existing office that was internet ready. We have many of our micro branches connect via VPN back to our main office. This worked find in all of our offices, except the new one. It turned out that something odd was happening to VPN connections from this location (and from other NATted locations, like coffehouse WiFi). Windows 2000 VPN clients had no difficulty, but XP clients failed to connect. For whatever reason, whenever XP clients tried to connect, GRE packets would get dropped by our SonicWall Pro firewall. The SonicWall doesn't pass GRE and couldn't be upgraded to support it (what a gyp). What was most annoying to me about it was that I couldn't fix the problem by merely applying a patch and making a registry change to XP (as should be expected). God, does M$ piss me off sometimes. Come to think of it, almost all the time.


So I'm fed up. There's no way I want to pay for another SonicWall-alike product if it's gonna be obsoleted several years later by some new protocol. What do I do? I'd played with IPCop on my home network several versions back when I had less confidence in its enterprise abilities. Specifically, I knew I needed multiple WAN IP address abilities (aliases). To my delight, I found that IPCop 1.4.8 has an IP alias option! Huzzah! I can use IPCop as our enterprise firewall!

So after a day of configuring the firewall, migrating settings, tweaking config, installing some addons, then removing some (GUIListenPorts is nice, but it needs to be more versatile), then a few hours in the evening testing it, I was running our 100+ employee network on IPCop and the VPN issue was fixed. Although IPCop is fairly easy to set up and use, the setup wasn't exactly simple in my case. I had some modifications I needed to make. I had to alter the admin web interface ports manually. HTTPS port 445 is blocked by many ISPs and port 443 was already in use for our webmail. GUIListenPorts really needs to allow the user to modify the port numbers by hand. It should suggest 80/443, but linux is all about configurability. Make it configurable as well as easy. I also had multiple subnets on the GREEN network. IPCop doesn't handle these by default, and can't be easily (web admin) configured to do so. I ended up having to add a static route entry in rc.local (/sbin/route add -net 10.0.0.0 netmask 255.255.0.0 gw 10.0.0.1). But that was only half of it. IPCop doesn't allow you to set a wider subnet or multiple subnets on GREEN, so NAT was broken on the remote subnets when talking to aliased RED IPs from remote subnets on GREEN. I had to fix that by changing a line in "/etc/rc.d/rc.firewall" from

/sbin/iptables -t nat -A POSTROUTING -m mark --mark 1 -j SNAT --to-source $GREEN_ADDRESS

to

/sbin/iptables -t nat -A POSTROUTING -s 10.0.0.0/16 -m mark --mark 1 -j SNAT --to-source $GREEN_ADDRESS

But wait! There's more! So after all that, things seemed to be working famously. Until another microbranch called. Now their PCs can only stay connected to the VPN for a little over 3 minutes. Something's not right, and it's IPCop's fault. At least that was my initial thought. I looked at the conenction logs on our Win2K RAS server and that said that the client disconnected gracefully. But the client obviously thought it was still connected until a minute or two later. WTF? Something is suddenly blocking packets, and it's not IPCop. So I drive to the second microbranch and watch this for myself. After a few hours of messing around, and upgrading the Netgear FWG114P, I noticed that the Netgear isn't forwarding the return GRE packets. I don't know why, but sopmething about adding the IPCop box on the main office network broke the way the VPN connections were being set up. GD! Proprietary hardware strikes again. I just wished the branch manager would have bought a Linksys WRT54G or WRT54GS instead. But wait! all I have to do is replace this P.O.S. with another IPCop box! IPCop comes to the rescue twice in as many weeks. But this wasn't a simple setup either. I had a ton of old Dell GX150's just begging to be turned into IPCop boxen, but the internal NIC wasn't auto-recognized by IPCop. Nor was the nice, cheap USR NIC (USR 7900, aka, USR997900 [two LEDs]) that I just bought. After a few hours of fiddling, I got it. You have to specify the module name in the /var/ipcop/ethernet/settings file. For the onboard NIC, use this...

GREEN_DRIVER=3c59x
GREEN_DISPLAYDRIVER=3c59x

For the USR NIC, use this...

RED_DRIVER=8139too
RED_DISPLAYDRIVER=8139too

You may have to set up the box to use dummy interfaces just to get past that part of the setup and get your initial settings file created.

After I got past that, I just reconfigured the now almost useless Netgear box to just act as an access point. Certainly not worth its initial price.

I should make one more comment about all of these IPCop modifications. Be careful. Keep track of all these mods as you go along (that's why I'm writing this actually). Although IPcop does have a settings backup function, it does not back up the specific files that I've modified (with the exception of ethernet settings). If you ever have to re-install it, or even if you upgrade it, these files may/will get overwritten, and you'll have to re-do all of your modifications. It's a pain, I know. It's the price you pay for an otherwise rockin' operating system. Maybe they will fix this sometime by maing these sorts of options available natively.


That's about it. Maybe if I'm adventurous, I'll try linking the IPCop boxen with local VPN connections (instead of using RAS) and tie our systems together in a cleaner manner. Maybe later, when I have free time. Much later :-)