Category: MuchTall's Linux Tips

Firstly, let me strongly recommend JDiskReport if a GUI is available to you. It's super easy to use and helps you quickly drill down into the disk and identify disk space usage, on any platform with Java support.

In lieu of that, if you have need to identify disk usage via the command line, run this set of commands:
find / -exec du -ks {} \; 2> /dev/null | sort -n | awk '{printf $1 "\t"; if (system("test -d \""$2"\"")) { print $2 } else { print $2 "/" } }' | tail -1000

This will spit out a list of the largest 1000 individual files and folder sums, sorted by size.

Depending on the size of the disk, this will take a while, as should be expected as it runs a "du" for each path it finds. Caching helps if/when you re-run this.

I don't know why this hasn't already been posted all over the interwebs, so I thought I'd post it here. It's amazing how some of the simplest answers never seem to make it into prominence in the Google search rankings.

Anyway, after cloning a VM today in Xen, I tried to start the DomU, and got this error:

Error: (12, 'Cannot allocate memory')

I assumed it had something to do with my Dom0 memory cache consuming nearly every last bit of free memory on my system, so I ran this to clear the cache:

echo 3 > /proc/sys/vm/drop_caches

I confirmed that that "free" reported that the cached memory dropped near zero. Great! Except it didn't help. I then ran "xm info" to check the memory usage in Xen

total_memory : 16378
free_memory : 2

Odd. I checked the Dom0 status to see what's up, and sure enough, my memory allocation on Dom0 is way high.

xm list Domain-0
Name ID Mem VCPUs State Time(s)
Domain-0 0 10482 4 r----- 3273110.4

But I still didn't know how to shrink it. I knew it had something to do with memory ballooning. I also knew you could set the memory allocation of a DomU with "xm mem-set", but I didn't know anything about how to do it with the Dom0. A quick IRC message to my Xen guru, Jima, and I get this simple command (obvious really) in response:

xm mem-set Domain-0 512

This balloons the Dom0 host system (as well as any domain, really) memory usage down to 512MB. In my case, my Dom0 claimed to be using about 2 Gig already, so I ballooned down to 4G instead. Not sure if that was just due to some bad math or assumptions on my part, as I never run anything of significance in Dom0. After you run this, you should be able to start your DomU just fine.

First, a little background...

The classic procedure for installing drivers to a printer shared via CUPS with a raw Samba frontend, usually is roughly something like this:

1. Create printer in CUPS
2. Open shared printer properties in Windows
3. Install drivers via Advanced tab

For me, this worked great, most of the time, assuming that the driver was either PS or PCL5(e). However, PCL6 drivers rarely loaded properly. They crashed upon upload, and the driver rarely ever worked, if point 'n print even installed the driver. As I understand it, the root of the problem is that the drivers need a valid "form" database specific to the driver to be stored on the server in order to load, but valid form data can only be created by executing the driver. On Windows servers, this isn't a problem, but Samba servers cannot execute the windows driver code in order to generate this form. PCL6 drivers are particularly finicky about this, and often times will refuse properly copy to the server.

The solution I've come up with is to load the driver entirely on a Windows PC first, and tell Samba to copy the drivers and related forms, ACLs, etc over. Some of the values I use here for the printer names, etc, are very generic and only specified to match the naming standards in my environment. Feel free to change your printer setup to suit your needs.

The Procedure
On a Windows PC:

1. Install the printer as you would normally for TCP/IP printing on a windows PC.
2. Rename the printer to the share name you will be using (i.e., CopierCP5000)
3. Open the printer properties
4. Share the printer using the same name (CopierCP5000)
5. Set the printer description to the printer name (i.e., Copier CP 5000)
6. Set the printer location to the office name and city (ACME Abu Dhabi)
7. Usually under the Accessories/Options tab, update/get information from the printer
        This ensures that we obtain all the correct hardware configuration information for the device (duplexer, trays, hole punch, stapler, etc)
8. Open the Advanced tab and click the "Printing Defaults" button
9. Click on each tab and allow the tab to load
        This ensures that the form values for each tab are created.
10. Still inside the Printing Defaults, change the page orientation to Landscape, hit apply, change the page orientation to Portrait, and hit apply. Click OK.
11. Close the printer properties by clicking OK

Next, on the server:

### Set up the printer in CUPS, i.e.:
lpadmin -p CopierCP5000 -L "ACME Abu Dhabi" -D "Copier CP 5000" -v lpd://192.168.xx.xx -m raw -E
smbcontrol smbd reload-config
 
### Copy the drivers, forms, ACLs, etc from the Windows PC
net rpc printer MIGRATE ALL CopierCP5000 -S mypcname -Uusername


Finally, test the driver in Windows:

1. Open the local Printers control panel (shortcut command below for Win7 users)
        rundll32 shell32.dll,SHHelpShortcuts_RunDLL PrintersFolder
2. Delete your shared printer
3. Right-click in the control panel window and select "Server Properties"
4. Open the Drivers tab
5. Select your driver and click "Remove".
6. Proceed with removal and close the server properties window.
7. Open \\servername\
8. Double-click your printer
9. Print a test page
        If this fails, try restarting the local PC's Print Spooler service

So far I've only tested this on a few of our known-problematic printers that have PCL6 drivers available, and it's worked perfectly in each case. However it you notice this doesn't work for you, post a comment and I'll see what I can do to help.

First, use conntrack to correctly identify your entries:

    conntrack -L -s 172.16.1.45 -d 123.123.123.123

This should display any connections that came from the internal IP of 172.16.1.45 destined to 123.123.123.123

Once you have confirmed the connections shown are the ones you with to delete/reset, paste the following after the command from above:

    conntrack -L -s 172.16.1.45 -d 123.123.123.123 | sed 's/=/ /g' | awk '{print("conntrack -D -s "$6" -d "$8" -p "$1" --sport="$10" --dport="$12)}'

This will print a list of the commands that would run to delete the connections. Replace "print" with "system" to execute the deletions:

    conntrack -L -s 172.16.1.45 -d 123.123.123.123 | sed 's/=/ /g' | awk '{system("conntrack -D -s "$6" -d "$8" -p "$1" --sport="$10" --dport="$12)}'

Finally, re-run the list command to see that all the entries have been removed:

    conntrack -L -s 172.16.1.45 -d 123.123.123.123

And you're done!

It's infrequent enough that I have to do this that I forget how. So to help me remember, I'm posting it here:

To make your permissions changes to devices in /dev/ survive reboots and become permanent, you need to alter the device permissions in

    /etc/security/console.perms.d/50-default.perms

So for example, say I have a created a group that holds all users on the system that are to have RW access to the video capture hardware (such as mythtv and zoneminder), open the perms file and look for this line:

    0600 0600 root

This v4l class covers all /dev/video* devices. To make them RW by the group "video", change the line to look like this:

    0660 0660 root.video

You can reboot to see the effect.

If you've ever tried to parse the output of "uptime", you know it can be complicated. There's a simpler solution: Use the raw numbers from /proc/uptime. Here's a simple script that spits out the days and remaining time in digital clock format:

    cat /proc/uptime | awk '{printf "%d%s%.2d%s%.2d%s%.2d%s" , $1/86400 , " days, " , $1%86400/3600 , ":" , $1%86400%3600/60 , ":" , $1%86400%3600%60 , "\n"}'

Note that the output can be changed to your needs. See the AWK Tutorial for more info on using modulus (%) in calculations, and string formatting (i.e., %d and %s)

If you've got a crashed disk, dd_rhelp is a very
useful tool for quickly recovering a large amount of data from just
about any failed disk. I used it earlier this week to help recover a
160GB linux LVM group member drive. Unfortunately, after a while of
running, it hit this error:

    dd_rhelp: error: add_chunk : invalid argument '0-(info)' (is not correctly formatted as number:number)

Here's
how you get past that. The problem is that dd_rhelp was not able to
determine the size of your hard disk automatically, so it is scanning
beyond the actual size, which of course fails. We need to tell it our
disk size manually.

First, make a backup of your logfile, for example:
    cp -rp sdb.log sdb.log.bad-eof
That way you can start over if you mess up.

Go to the first line in the original log file that states:
    dd_rescue: (warning): /dev/sda (282662620.0k): No space left on device!
Or something similar. Scroll up to the next line before that that reads:
    === COMPUTED VERSION OF LOG :
Delete all lines including and following this line, then save your file.

Obtain your filesystem size with fdisk:
    # fdisk -l /dev/sda
   
    Disk /dev/sda: 160.0 GB, 160000000000 bytes
    and so on...

In this case, your filesystem size would be 160000000000 bytes, or 160000000  kbytes.

Next,
change all the lines in the file stating "eof:nothing" to
"eof:160000000". Use your fs size in replacement of 160000000 kbytes.
You can quickly do this using sed:
    sed 's/eof:nothing/eof:160000000/g' -i sdb.log

Now re-run your dd_rhelp command. It should now continue to recover as normal!

Thanks to Valentin from dd_rhelp and "Master One" for leading me in the right direction.

UPDATE:
It seems that after a short time, I ran into the same error again. My
solution was to open the dd_rhelp script, look for the line that says:
    eof="nothing"
and change it to
    eof="160000000"
(of course, substituting your disk size in kb)

Re-do the cleanup above and re-run.

UPDATE 2: So I've found that dd_rhelp is slow, or at least, it's slow on my damaged disk. I suggest trying ddrescue instead (not to be confused with dd_rescue from above). It seems to be much more efficient at sidestepping bad data and gives a more concicse summary of recovery progress. This includes how much data has been recovered so far, lost so far, number of total errors, and transfer rates. For more info on the design differences between ddrescue and dd_rhelp, see the SpinRite wikipedia entry.

The default setup for Hylafax is to drop faxes into an email and send them off to the "FaxMaster". This weekend I needed to stop hylafax from doing that, and instead just drop it into a folder. Best I can tell after a few hours of searching Google and perusing code, hylafax on it's own does not do this. There appears to be no options for it in faxrcvd.

So instead of searching any longer, I decided to write my own, very simple, yet perfectly funtional faxrcvd script. All it does is take the received TIFF image and converts it to PDF (the preference of my users), and drops it into a specified folder. Here's the code, feel free to use it:

# Make a backup of the original fax received script (faxrcvd emails faxes).
cp -p /var/spool/hylafax/bin/faxrcvd /var/spool/hylafax/bin/faxrcvd.orig

EDIT: I received a message the other day saying that my script didn't work. It's possible some of the code got messed up by Nuke, so I'm just suggesting downloading instead...

# Download the MuchTall faxrcvd code in replacement of the existing code
wget -O - /"content/faxrcvd > /var/spool/hylafax/bin/faxrcvd

# Edit it with your favorite editor to change the path and permissions
vi /var/spool/hylafax/bin/faxrcvd
    OR
nano -N /var/spool/hylafax/bin/faxrcvd

Change the path and permissions (chmod) to what you want. In my case, I needed to allow all my users R/W access to these faxes.

There you go! Assuming you already have configured hylafax, you should be all set!

When setting up PHPNuke, you really have to remember to turn on captcha checking for registrations first thing in config.php (gfx_chk). If you don't, you'll get pummeled with comment spam in a matter of hours.

If, like me, you messed up and forgot to, you'll have to delete those bogus accounts, and delete the comments. Deleting the comments can be tedious. So instead, run a command from a unix shell to generate some html content that will make your life easier...

mysql -u dbusername -pdbpassword dbname -e 'SELECT tid,sid FROM nuke_comments;' | grep -v tid | awk '{print "<a href=\"http://yourdomain.com/admin.php?op=RemoveComment&tid=" $1 "&sid=" $2 "&ok=1\">http://yourdomain.com/admin.php?op=RemoveComment&tid=" $1 "&sid=" $2 "&ok=1</a>"}'

This will output a bunch of HTML that you can insert into a page. Click on each of those links to delete a comment. Much quicker than searching for them all.

Now, I suppose someone could write a PHP module for Nuke that would automate this. I'm just too lazy to ;-)

I just got Tiny MCE updated on MuchTall and I just thought I'd pass the update instructions on

1) Download and unpack Tiny MCE from http://tinymce.moxiecode.com/download.php
2) Move the folder /includes/tiny_mce to /includes/tiny_mce.orig
3) Copy the new tinymce/jscripts/tiny_mce folder to /includes/tiny_mce
4) Download this javascript.php.patch patch into /includes/
5) Run "patch -p0 < javascript.php.patch" from within /includes/
6) Open your PHPNuke editor page

That's it! Now you have a rich text editor with advanced editing features right within PHPNuke!

A buddy of mine was recently setting up bind on his linux box and had an issue. His zone files wouldn't load without spewing out errors:

Jun 8 10:33:05 server named: hisdomain.com:18: unknown RR type 'host1'
Jun 8 10:33:05 server named: hisdomain.com:19: unknown RR type 'host2'
Jun 8 10:33:05 server named: hisdomain.com:20: unknown RR type 'host3'
Jun 8 10:33:05 server named: hisdomain.com:23: unknown RR type 'host4.hisdomain.com.'
Jun 8 10:33:05 server named: dns_master_load: hisdomain.com:24: isc_lex_gettoken() failed: unexpected end of input
Jun 8 10:33:05 server named: dns_master_load: hisdomain.com:24: unexpected end of input
Jun 8 10:33:05 server named: hisdomain.com: file does not end with newline
Jun 8 10:33:05 server named: zone hisdomain.com/IN: loading from master file hisdomain.com failed: unknown class/type
Jun 8 10:33:05 server named: _default/hisdomain.com/in: unknown class/type

Fun stuff. After messing around with his domain files for about an hour, I realized something. There were all these excessive spaces after each line. So I opened vi and proceeded to remove them all, as well as replacing spaces in the doc where a tab should be. Lo and behold, it loaded fine.

It appears the the root of the problem were those "spaces" at the end of the line. I figure they were probably DOS carriage returns created my some lame windows text editor. Once removed, the file loads fine.

I recently had to re-install my PC and I decided to take the opportunity to freshen up my Firefox profile by abandoning my prefs.js file and only migrating settings and extensions as needed.

As part of this, I ran into an issue I dealt with a while back and had to look up once more.

http://ffextensionguru.wordpress.com/2006/10/22/firefox-20-close-tab-buttons/

I don't like having my little red X to close the tabs actually in the tab. I like it at the end of the tab-bar. Just a quick rundown of how to change this:

Browse to about:config and filter for browser.tabs.closeButtons. Set that value to 3.

Disclaimer: Nevermind. Turns out that this only disables lookups on remote syslog entries (from another syslog host). Still looking for a solution on this one. Feel free to read on if you wish.

So this weekend, someone (as happens quite frequently) attempted to gain access to my server via the FTP daemon. No big surprise there. What was interesting about it was that there was one attempt from an rhost named "oa". Not oa.1337hacker.com, or even an IP, just.... oa. As expected an nslookup of "oa" yielded nothing.

So I ended up turning off reverse lookups on syslog. I'd suggest you do the same. You can't really trust reverse lookups. What's to prevent someone who has control over their PTR record to create any bogus reverse lookup, even google.com or yourdomain.com and attempt the same thing? You have no means at that point to actually identify the attacker positively by IP.

To turn off reverse lookups for syslog on FC6, just add -x to this line in /etc/sysconfig/syslog:

SYSLOGD_OPTIONS="-x ... "

See here for my bug report on Redhat's Bugzilla. https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=227357

Some dumbass has been pummeling my ftp server with brute-force FTP attempts. So I've written a script that you can add to your cron tab to help you auto-ban them via hosts.deny. Pay attention to the variables as this was written on a RedHat 9 box and may not be entirely applicable to other platforms or versions.

ftpautoban.pl

Just recently I got OpenVPN set up on my WRT54G (w/DD-WRT) to connect to the network at my workplace. However, I noticed that although the vpn tunnel was up, packets weren't traversing it. For a while I thought it was some odd routing issue, but then I realized that the issue was with the firewall config. I just assumed that the firewall config was automatically changed to allow VPN tunnel traffic.

So here's the problem. DD-WRT is meant, on it's face, to be a cutesy interface for average Joes, with some power-user features. I doesn't allow you to set custom firewall rules via the web interface. Furthermore, there's no flat file you can edit within the console to make changes. All configs get regenerated and overwritten on boot. The configuration alterations are actually stored in nvram. To see this data, just ssh to your DD-WRT and enter:

nvram show

Ahh. Brings back memories of configuring SpeedStream routers from scratch. Anyhow... Here's where you can see all the config that the DD-WRT uses to actually generate the volatile standardized conf files that sit in the ramdisk. You can actually see individual configuration variables if you'd like to alter or add to them by hand (instead of by web). For example, this will show you the NAT forwarding config:

nvram get forward_spec

You should see a list of strings showing the NAT config. It's all on one line with options separated by colons (and a ">") and delimited by spaces. That is, assuming you have any NAT entries configured.

But here's the fun part. I have 2 iptables rules that need to be inserted when the system boots to allow the VPN tunnel's traffic to pass:

iptables -I INPUT -i tun+ -j ACCEPT
iptables -I FORWARD -i tun+ -j ACCEPT

First, check to make sure you don't have any existing config that you may have to add:

nvram get rc_firewall

I can add this to the startup config by sending following commands:

nvram set rc_firewall="iptables -I INPUT -i tun+ -j ACCEPT
iptables -I FORWARD -i tun+ -j ACCEPT"
nvram commit

Copy this into notepad, edit it as you see fit, and paste it into your SSH session. After running them, just reboot to apply the change.

UPDATE 11/17/06: Apparently my habit of making solutions geekier than they have to be has bitten me again. It appears that if you simply browse to Administration > Commands (http://yourrouter/Diagnostics.asp), you can enter the commands there and hit "Save Firewall" to achieve the same effect. Damn. And here I thought I discovered something cool :-)

I recently updated Muchtall.com to serve RSS and Atom newsfeeds for the site. For more info on how to do this to your own PHP-Nuke site, click here:

RSS and Atom for PHP-Nuke

It's not a linux tip, but it's still a tip...

If you're looking for an easy way to keep Spybot S&D updated and your system immunized, try this registry file which launches Spyboy S&D, updates, and immunizes in the background on logon.

Spybot AutoUpdate and AutoImmunize On Logon.reg

A few weeks ago at my job, we started up a new mini-branch in an existing office that was internet ready. We have many of our micro branches connect via VPN back to our main office. This worked find in all of our offices, except the new one. It turned out that something odd was happening to VPN connections from this location (and from other NATted locations, like coffehouse WiFi). Windows 2000 VPN clients had no difficulty, but XP clients failed to connect. For whatever reason, whenever XP clients tried to connect, GRE packets would get dropped by our SonicWall Pro firewall. The SonicWall doesn't pass GRE and couldn't be upgraded to support it (what a gyp). What was most annoying to me about it was that I couldn't fix the problem by merely applying a patch and making a registry change to XP (as should be expected). God, does M$ piss me off sometimes. Come to think of it, almost all the time.


So I'm fed up. There's no way I want to pay for another SonicWall-alike product if it's gonna be obsoleted several years later by some new protocol. What do I do? I'd played with IPCop on my home network several versions back when I had less confidence in its enterprise abilities. Specifically, I knew I needed multiple WAN IP address abilities (aliases). To my delight, I found that IPCop 1.4.8 has an IP alias option! Huzzah! I can use IPCop as our enterprise firewall!

So after a day of configuring the firewall, migrating settings, tweaking config, installing some addons, then removing some (GUIListenPorts is nice, but it needs to be more versatile), then a few hours in the evening testing it, I was running our 100+ employee network on IPCop and the VPN issue was fixed. Although IPCop is fairly easy to set up and use, the setup wasn't exactly simple in my case. I had some modifications I needed to make. I had to alter the admin web interface ports manually. HTTPS port 445 is blocked by many ISPs and port 443 was already in use for our webmail. GUIListenPorts really needs to allow the user to modify the port numbers by hand. It should suggest 80/443, but linux is all about configurability. Make it configurable as well as easy. I also had multiple subnets on the GREEN network. IPCop doesn't handle these by default, and can't be easily (web admin) configured to do so. I ended up having to add a static route entry in rc.local (/sbin/route add -net 10.0.0.0 netmask 255.255.0.0 gw 10.0.0.1). But that was only half of it. IPCop doesn't allow you to set a wider subnet or multiple subnets on GREEN, so NAT was broken on the remote subnets when talking to aliased RED IPs from remote subnets on GREEN. I had to fix that by changing a line in "/etc/rc.d/rc.firewall" from

/sbin/iptables -t nat -A POSTROUTING -m mark --mark 1 -j SNAT --to-source $GREEN_ADDRESS

to

/sbin/iptables -t nat -A POSTROUTING -s 10.0.0.0/16 -m mark --mark 1 -j SNAT --to-source $GREEN_ADDRESS

But wait! There's more! So after all that, things seemed to be working famously. Until another microbranch called. Now their PCs can only stay connected to the VPN for a little over 3 minutes. Something's not right, and it's IPCop's fault. At least that was my initial thought. I looked at the conenction logs on our Win2K RAS server and that said that the client disconnected gracefully. But the client obviously thought it was still connected until a minute or two later. WTF? Something is suddenly blocking packets, and it's not IPCop. So I drive to the second microbranch and watch this for myself. After a few hours of messing around, and upgrading the Netgear FWG114P, I noticed that the Netgear isn't forwarding the return GRE packets. I don't know why, but sopmething about adding the IPCop box on the main office network broke the way the VPN connections were being set up. GD! Proprietary hardware strikes again. I just wished the branch manager would have bought a Linksys WRT54G or WRT54GS instead. But wait! all I have to do is replace this P.O.S. with another IPCop box! IPCop comes to the rescue twice in as many weeks. But this wasn't a simple setup either. I had a ton of old Dell GX150's just begging to be turned into IPCop boxen, but the internal NIC wasn't auto-recognized by IPCop. Nor was the nice, cheap USR NIC (USR 7900, aka, USR997900 [two LEDs]) that I just bought. After a few hours of fiddling, I got it. You have to specify the module name in the /var/ipcop/ethernet/settings file. For the onboard NIC, use this...

GREEN_DRIVER=3c59x
GREEN_DISPLAYDRIVER=3c59x

For the USR NIC, use this...

RED_DRIVER=8139too
RED_DISPLAYDRIVER=8139too

You may have to set up the box to use dummy interfaces just to get past that part of the setup and get your initial settings file created.

After I got past that, I just reconfigured the now almost useless Netgear box to just act as an access point. Certainly not worth its initial price.

I should make one more comment about all of these IPCop modifications. Be careful. Keep track of all these mods as you go along (that's why I'm writing this actually). Although IPcop does have a settings backup function, it does not back up the specific files that I've modified (with the exception of ethernet settings). If you ever have to re-install it, or even if you upgrade it, these files may/will get overwritten, and you'll have to re-do all of your modifications. It's a pain, I know. It's the price you pay for an otherwise rockin' operating system. Maybe they will fix this sometime by maing these sorts of options available natively.


That's about it. Maybe if I'm adventurous, I'll try linking the IPCop boxen with local VPN connections (instead of using RAS) and tie our systems together in a cleaner manner. Maybe later, when I have free time. Much later :-)