I just solved a weird MySQL replication problem and thought I would share with you all. First, the background. I have a master and slave running with one-way replication. The slave just sits by waiting for his time to shine, but otherwise doesn't do anything. Well, last week the master had a problem with the filesystem. I can't recall exactly what it was, out of space or something. It confused the heck out of the master and so it basically shut down. One of my co-workers fixed the problem and got the master running again, but the slave was in a pickle. Here is the error it was showing:

Relay_Master_Log_File: mysql-bin.031
     Slave_IO_Running: Yes
    Slave_SQL_Running: No
           Last_errno: 0
           Last_error: Query 'DELETE FROM foo WHERE bar = 1' caused different errors on master and slave.
                       Error on master: 'Got error %d from table handler' (1030), Error on slave:
                       'no error' (0). Default database: 'baz'
  Exec_master_log_pos: 118871

Because it thought the command failed on the master, it refused to continue. I can't say it's an altogether bad plan since data integrity is generally the main theme of a database (yes, cue the jokes about my using MySQL in the first place). The question became, how do I get the slave to start up again. "SLAVE STOP; SLAVE START" didn't have any effect.

The trick was suggested to me by a post at mysql.com which pointed out a tool new to me, mysqlbinlog. See I figured the simplest thing would be to restart replication at the step just after the "failed" transaction, since I knew that transaction had actually succeeded. But I have no idea how the binlog counters work, so I couldn't just make up numbers. It's some kind of binary offset. Well, mysqlbinlog will show it to you.

# mysqlbinlog mysql-bin.031 -j 118871 |less

Which of course showed me this:

# at 118955
#080605 18:59:09 server id 1  log_pos 118955    Query   thread_id=3218  exec_time=0     error_code=0

So on my slave I restarted replication at offset 118955 and like magic, the slave ripped through the binlogs and caught up in practically no time at all.

Hans and I were discussing QoS the other day, specifically regarding using Wondershaper from the LARTC. I had managed to mess mine up and I subsequently noticed a horrible turn for the worse in my VoIP calls. Wondershaper has to be adapted for use by OpenWRT and in the process I misspelled sch_ingress.o as sch_insmod.o. Too much insmodding that day, I think. The net effect was that download speeds were not shaped at all.

Once I got it corrected, I decided to do a few tests just to confirm that using Wondershaper actually made a difference. I'll cut to the chase for the lazy: it did. I made 45 second calls to music on hold from my softphone, Twinkle. In the background I had Wireshark running. I used the RTP analyzer in Wireshark to look at the statistics after all was said and done. I used both versions of Wondershaper, the CBQ and HTB. I had a single download running the whole time eating up all spare bandwidth.

With no shaping: 4.4% loss (95 packets), 60ms jitter
With CBQ Wondershaper: 0.2% loss (5 packets), 35ms jitter
With HTB Wondershaper: 0.3% loss (6 packets), 28ms jitter

So my unscientific conclusion is that both versions of Wondershaper work about the same and they both make a huge difference. I could easily hear the packet loss on the first call, but not so much on the other two calls.

I ran across a site today named Leads Online. It's a tool for law enforcement to collaborate with pawn shops and the like. Essentially the pawn shop uploads all of its transactions at the end of the day and then if a law enforcement agency is looking for some stolen stuff, they search through the database. Leads Online tells how it's such a convenient and easy service, and helps good guys catch the bad guys.

My first thought was substantially different. I don't really relish in the thought of law enforcement having carte blanch access to business records, possibly without warrants. There's some good potential for abuse there. I recently read through all the Idaho Falls city ordinances (yes, it was actually interesting) and I'm aware that pawn shops are required to keep records of purchases for 2 weeks (IIRC), just in case a question of ownership arises. But there is not a requirement for them to proactively send those details to the police department. That seems to be a fair trade.

One immediate concern I have is that being a private company, they are not subject to laws that courts and police would be. They don't have to answer Freedom of Information Act responses. They aren't subject to public oversight. I don't think we should be outsourcing our key public infrastructure to private industry whose primary motivation is the almighty buck.

Even more than that, what really scares me isn't the intended uses but the ways the system could be expanded. There are all sorts of ways the database could be reused for marketing purposes, thus invading the privacy of sellers and buyers alike. There are pathetic privacy laws in the country, so it's hard to believe that anyone would have any recourse.

And what if law enforcement decided they wanted to browse through the database? The obvious abuse would be looking for religious material. Yeah, it seems pretty unlikely but the point isn't that we trust good people. I know a few law enforcement people and I would have no qualms with them using something like this. The point is that just as there are unscrupulous citizens, there are unscrupulous police. We need to make sure the system fails gracefully and I don't believe this one does.

Generally speaking I find most of what Mark Cuban has to say fairly good, but today I have to disagree. His latest rant against P2P I think starts off with faulty premises, thus the conclusions don't hold.

As a consumer, I want my internet experience to be as fast as possible. The last thing I want slowing my internet service down are P2P freeloaders. Thats right, P2P content distributors are nothing more than freeloaders. The only person/organization that benefits from P2P usage are those that are trying to distribute content and want to distribute it on someone else's bandwidth dime.

Have we conclusively shown that P2P slows down the Internet? I'd like to see the proof of that. It's apparent that what he's mainly complaining about here is the amount of upload traffic. If somebody downloaded the same content via non-P2P, aka the non-freeloading way, they would eat up the same amount of download. So the only difference is the extra upload with P2P. Is that really a problem though? I can tell you, my network has upload to spare. I would estimate that our upload traffic is about half of what the download is at.

Does anyone really think its free ? That all the bandwidth consumed with content being distributed by P2P isn't being paid for by someone ? That bandwidth is being paid for by consumers.

When I buy a DS3, I get 45mbps of bandwidth in each direction. I have to have enough to support the download demands of my customers and I just get the upload to boot. It's just sitting there. So yeah, the consumer is paying for it but there's no way to recover that cost. Even if every consumer stopped uploading, they would still pay the same amount. Why not put it to some good use?

And there is a good use. If that bandwidth isn't used by the P2P content distributors, they end up forking out additional money on their end for the extra bandwidth. They then have to pass the cost on to consumers. Why should consumers pay twice?

Consumers who pay for personal, not commercial applications. When consumers provide their bandwidth to assist commercial applications, they are subsidizing those commercial applications which if it isn't already, should be against an ISPs terms of service.

I'm a little surprised honestly, to see Cuban talk about consumers as people who just eat and eat whatever the big media companies feel generous enough to feed them. What about the generation of the long tail? Consumer created content and all that jazz? I realize that most of what's going on with P2P is unauthorized distribution of other peoples' copyrighted works, but baby, bathwater. You know the saying.

And finally, to round out my argument I will point out that blocking P2P isn't an easy task. P2P software designers have made is hard on purpose. That means ISPs have to shell out for big hardware to do the task. Who ends up eating those costs? I would have to redesign my network in order to even accommodate the devices, which would add even more.

I won't even get into the whole neutrality debate. That's a can of worms for another day.

Back in July I went to the Idaho Department of Transportation's website with the intent to renew my car registration. I was rather surprised that the connection was "reset by peer". OK, their site is having trouble. I'll just come back later, right? Next day, same thing. That's when I got suspicious. I played around for a while and discovered that by changing the browser identification string, I could get in. Well, that was good because I was rather uninterested in getting arrested for driving illegally.

Once things settled down, I wrote to their webmaster and complained. I was a little rude the first time, yammering on about tax dollars and discriminating. I knew that would get their panties in a wad, and it worked. They responded and asked for clarification. I provided them with quite detailed info (maybe too detailed) and offered to work with them to get it resolved. They never wrote back and I forgot about it.

Well flash forward to this evening when the same thing happened again, this time on Idaho Public Television's website. I knew it couldn't be a fluke that I'd get the same reaction with the same workaround. One difference is that I've been to Idaho PTV's website on this computer before, using this same browser, so they must have changed something on their end to cause it.

First I eliminated Cold Fusion which Idaho PTV is using but IDT isn't. Then I looked at the web servers, Idaho PTV is IIS 5.0 and IDT is IIS 6.0. Well I had assumed they must be running IIS because what other crack pot web server out there would do something so inane?

Finally I stumbled upon the perfect test: grab a capture of the headers that Firefox sent and make slight alterations until I figured out the exact character or combination of characters to break their site. And that is what I did. I saved the headers to a text file and piped them to netcat, that most useful of network tools. Here is the browser string as it appeared unaltered:

User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20061201 Firefox/2.0.0.6 (Ubuntu-feisty)

Starting from the end, I removed "(Ubuntu-feisty)" from the browser and tried it out. Voilà! That was the offending part. Lucky for me I got it on the first try.

I narrowed it down by process of elimination to the four characters "isty". They can appear anywhere in the User-Agent header and it will immediately cause IIS to send a TCP reset. Not even an HTTP error code, but a RST. I tried it in other headers and there was no problem (e.g. X-Linux-Distro: Ubuntu-feisty).

The only sense I could make out of this behavior is some sort of security setting in IIS. It's doing some sort of content analysis and determining that anybody who uses the letters "i", "s", "t" and "y" in the User-Agent header is a bad guy. With logic like that, you'd think the IIS team was working for the TSA. Hmm, a conspiracy maybe? Anyway, I remember when I was a lowly IIS admin that there was some security lockdown tool that Microsoft recommended. I wonder if that's what's doing it. Or maybe it's an antivirus software. It's hard to say. I think I have a good lead with Idaho PTV since it just started happening. Surely they must remember the changes they've made to their production web server. I might just write to them and find out.

In the mean time, I smell some sort of nefarious hack here but I can't come up with anything good. Somehow you've got to be able to leverage this bug to bring doom upon unsuspecting IIS users. If you've got any ideas, please post them in the comments.