apt-get update Fails Due To Gzip Error

This morning I was greeted by a collection of apt-get failures. I use cron-apt on all my servers so I can quickly apply patches. A few of my servers reported the following error when they ran last night:

gzip: stdin: not in gzip format
Failed to fetch http://http.us.debian.org/debian/dists/etch/main/binary-i386/Packages.gz  Sub-process gzip returned an error code (1)
E: Some index files failed to download, they have been ignored, or old ones used instead.

A little googling revealed that it's some sort of bug in apt. The solution was pretty easy, simply remove all the files in /var/lib/apt/lists/partial.

The oddest thing about this issue is that it struck multiple servers of mine on the same day. Seems there must have been something on the Debian servers that triggered it. Maybe the web server crumbled and reset connections, leaving a partial file? That would be my guess but I'll probably never know for sure.

tags: 

Cable One Browser Hijacking

I started seeing alerts today which said the following:

CableONE is excited to present the new in-browser notification system!

CableONE wants to keep you informed about critical service changes, maintenance events and important account information on a more timely, and therefore useful, basis. In order to do so, CableONE may periodically send you bulletins, like this one, which will automatically appear within your Internet browser. This process does not involve collecting any information about your activity on the Internet and, other than this initial communication to receive your preferences, will be a one-way process. To learn more about this new communication method, as well as how to change or configure your notifications, please click the “Learn More” button located above. Thank you!

I immediately opted out of the system as it frankly creeps me out that they're hijacking my HTTP traffic for whatever purposes they want. I can see the temptation. They're planning weather alerts and amber alerts. They want to alert me to network issues which may be affecting me. I get the benefits. Call me old school, but I want an ISP that just connects me to the Internet and leaves my traffic alone.

I also worry about the precedent. Maybe they start sending me notifications of really good deals from their partners. Maybe they start changing banner ads to be from companies they select. Perhaps they start blocking sites they don't like, or that don't pay them a fee. It's a trend I wouldn't like to see started.

Bottom line, if I wanted somebody mucking with my bits, I'd sign up with Comcast.

tags: 

Shaky TV Playback in MythTV


I joined the digital TV revolution yesterday with the purchase of a Vizio 37" 1080p HDTV. So far we are quite enjoying it. One issue that is still ongoing is that the TV's remote overlaps with the DVD player's (which we are also using for the MythTV). When you push the Zoom, number or volume keys you get more than you bargained for.

But that issue is ongoing. The one I'm writing this post about has been solved, thanks to a mythtv-users post I came across. The symptoms were that some video, for me it was anything on CBS or ABC but curiously no others, would be really shaky. It was almost like an interlaced shearing effect, but not so regular or consistent. I played with the interlacing options and output drivers to no end. Finally I came across the above thread and discovered the problem: a bug in the nVidia kernel driver. I simply upgraded to the latest version and the problem went away. This is a very good thing.

Otherwise everything about the MythTV setup has worked well in high-def. I just had to add a "1920x1080" mode in xorg.conf and then do the same in MythTV. I later adjusted MythTV to use fullscreen, regardless of the resolution, which is what it should have been in the first place. I was worried that it might take some major tweaking to get widescreen or high-def or who knows what else, so overall I'm very pleased.

tags: 

HDHomeRun Power Supply

I sadly discovered today that nothing on my MythTV was recording, and even live TV was not viewable. "What gives?" I pondered as I poured through the logs.

# tail -f /var/log/mythtv/mythbackend.log
2010-02-21 22:06:29.747 TVRec(2): Changing from None to WatchingLiveTV
2010-02-21 22:06:29.757 TVRec(2): HW Tuner: 2->2
2010-02-21 22:06:30.768 HDHRChan(xx/1), Error: device not found
2010-02-21 22:06:31.780 HDHRChan(xx/1), Error: device not found
2010-02-21 22:06:31.782 HDHRChan(xx/1): SetChannelByString(6-1), Error: Channel object will not open, can not change channels.
2010-02-21 22:06:31.783 TVRec(2) Error: Failed to set channel to 6-1. Reverting to kState_None

Well, that didn't bode well. I sauntered into the data center (the nook under the stairs without adequate power, lighting or cooling) and found the power light on my HDHomeRun was blinking and worse, so was the light on the power supply. A quick google turned up a known failure with the power adapters on some versions of the HDHomeRun. Guess who was lucky enough to receive one of them?

So this post is just an FYI to anybody else who bought an HDHomeRun. The RMA process is dead simple, just enter your device ID, and they'll send you a replacement for free. But the question is, what are we going to do for 3-7 business days while we wait? Back to live TV again? Oh the humanity!

tags: 

Math Fun

Saw a post at badscience.net which piqued my interest. Before you read it, first think about the odds of a mother having 3 children on the same day of the year. How likely (or unlikely, as the case may be) would you guess it is? Turns out it's a lot more likely than most people would think. OK, now you can click the link.

FWIW, thanks to my recently revived statistical knowledge, I did pick up that the correct odds wouldn't be 365*365*365, but rather just 365*365. That's because once the first kid is born, the chance of him being born on that day is exactly 1 (unless she's a Chinese gymnast in which case she'd be born 3 years later). Think of that the next time someone asks, "what are the chances of that happening again?". The answer is, the same as the chances that it happened in the first place (assuming the factors haven't changed, of course).

I also realized that the odds of same day births would probably be lower due to the dates of conception being non-independent. For instance, my wife and I planned to have our kids in the spring because she didn't want to be pregnant during the heat of the summer. So the range of possible days our kids could have been born is only a window of about 100 days. Would you believe they were all born inside that window? (Actually, one of them did sneak in a bit late, but within a decent margin of error).

tags: 

Kamailio Hashtables

A few weeks ago I completed an upgrade from OpenSER 1.2 to Kamailio1.5. Overall it's been working quite well and actually was not nearly as hard as I feared it might be. Now that I've got this new version with a ton of new bells and whistles, I thought I'd try out a few of them. One of the most exciting is shared memory hashtables.

First a quick primer on hashtables. Currently whenever the proxy needs a value from the database it simply queries the database and gets the value. Straightforward, right? But as the system grows, the database can be greatly put under strain. In my Kamailio cluster the single biggest CPU user is in fact the MySQL database by a large margin. And since the database changes pretty infrequently it would be nice if we could cache those values.

The problem is that Kamailio is a multi-process daemon, so what's cached in one isn't going to be cached in another. Enter shared memory hashtables. The same cache is shared among all the processes. Usage is pretty simple. Here's an example:

# IP based authentication
if ($sht(ht1800=>$var(fU)::ipauth) == null){
	if (is_user_in("From", "ipauth")){
		$sht(ht1800=>$var(fU)::ipauth) = "yes";
	}
	else{
		$sht(ht1800=>$var(fU)::ipauth) = "no";
	}
	$var(ipauth) = $sht(ht1800=>$var(fU)::ipauth);
}
else{
	$var(ipauth) = $sht(ht1800=>$var(fU)::ipauth);
}

if ($var(ipauth) == "yes"){
	# do whatever...
}

Don't worry too much about the nitty-gritty. First we check to see if the value is cached (($sht(ht1800=>$var(fU)::ipauth) == null)). If it's set, we just use the cached value and continue. If not, we have to ask the database (is_user_in("From", "ipauth")) and then save the result in the cache. It actually is that easy.

But the $64,000 question, has it helped? Luckily I thought ahead and got some data before I made the change. I grabbed the query log for a day before and after the change. I made sure to select two days where the call usage was the similar (1% difference). Overall there was a 20% reduction in SQL queries.

I only put in the one change as a proof of concept. Since it sure seems to have panned out I'll continue wrapping all my database calls with hashtable caches. One thing I still need to work out is how to invalidate the cache when I make a change through my provisioning system. And once I'm able to upgrade to Kamailio 3.0 I'll switch over to memcached, which has all the benefits of a shared memory hashtable but it's also clustered. Ooh, I'm so excited!

tags: 

dpkg --configure -a

So, just a word to the wise: dpkg --configure -a is not the same as dpkg-reconfigure -a. The former will continue configuring packages where an interrupted session left off. That's a good thing. The latter will reconfigure every package on your system. That, I think I can safely say, is a bad thing.

And no, I don't know anyone who's made that mistake recently. Nope, nobody.

tags: 

An Electric Lighting Audit

A while back I did an energy audit in my house. Fortunately for me I do live in an area served by cheap hydroelectric and coal energy. Well, fortunately for my pocketbook anyway. Regardless, I'm ever attempting to reduce my energy needs for both personal and altruistic reasons, hence the audit.

I surveyed all the lights in my home, noted their location, type, wattage, etc. When we moved in, most of the lights were old school incandescents. Altogether they added to 2918 watts of lighting. By swapping in a few compact fluorescents I've reduced that total to 1778. That's a savings of 40%. Impressive, but not so fast. Are all those lights used the same amount? Definitely not. The garage lights are on for probably only 10 minutes per day, on average. So clearly we need to factor actual usage into the equation.

So I also estimated the average usage of each bulb. Yes it was as fun as you think. What I found is that our 2918 watts account for about 4950 watt hours of actual usage per day. After my upgrades that dropped to 3274 watt hours, for a savings of 34%. If you notice that's less than the 40% we calculated earlier you get a brownie point. Indeed some of the lights we use the most are ones that I cannot reduce any further.

Even more interesting is something I've recently come to understand as I make a concerted, although to many unquestionably bizarre, attempt at understanding statistics. Quite often a very small number of things will often account for most of your results. A good example is the US federal budget. While McCain notably complained about earmarks in bills (which may be undesirable for many reasons) they account for a very small fraction of the money spent. If you want to make any reasonable dent in the outlays your best bet is defense, social security and medicare/medicaid which account by themselves for 65% of our expenditures.

Getting back to the original point (and hopefully avoiding a political tangent), I found that just a few of the lights in the house account for the majority of the electrical usage. In fact, the two sets of fluorescent lights in the family room account for nearly 60% of the total electricity. Even if I were to upgrade the rest my bulbs to compact fluorescents, it would only decrease my current usage by another 15%. I'm pretty much bound by those two sets of lights.

The exercise was a useful one. I'm on a slow quest to analyze all the electrical usage in the house. The light bulbs were the easiest to tackle. Some time I'll move on to the appliances and computers. The latter I'm rather apprehensive about. I might rather be ignorant.

My data files:
OpenOffice.org format
CSV format

tags: 

OpenWRT and WPA

For whatever reason last night my OpenWRT access point decided to stop working. My laptop would connect for a second and then disconnect. I tried another wifi card and it did the same thing, which eliminated my laptop as the culprit. I turned off WEP and was able to connect. The weirdest thing was when I would try to turn WEP back on, I got this error (in bold):

[root@alberto ~]# iwconfig wl0
wl0 IEEE 802.11-DS ESSID:"zmonkey.org"
Mode:Master Frequency:2.437 GHz Access Point: 00:0F:66:4A:DF:08
Tx-Power:19 dBm
RTS thr:2347 B Fragment thr:2346 B
Encryption key:<too big>

I'm not comfortable without encryption (although honestly I would like to run a captive portal alongside my encrypted network), so after a few reboots and unsuccessful attempts to get WEP working I gave WPA a shot. It's really what I should be running anyway but I've had WEP set up for so long I just didn't feel like dealing with it.

But luckily, it worked out quite flawlessly. Following a guide I found here I got it set up quite simply. I did have to reboot to get the changes to take, but not a big deal. The only remaining problem is reliably getting my laptop to use WPA. That's the real reason I've delayed. I've gotten it to work before, but it's sometimes been a pain. All in the name of progress, I suppose.

tags: 

Pages

Subscribe to zmonkey.org RSS Subscribe to zmonkey.org - All comments