txspdy: Day 2

Fighting with C. You hate loving it. You love hating it. But like water, it’s good to know how to swim in it because 3/4 of the computing infrastructure is built on it.

I needed to build the openssl-1.0.1-beta2, then make a mod of pyopenssl to add in the NPN feature. Both of these went fine, and I got to a point that I had a clean compile. Trying to run a unit test showed me that something was segfaulting. Backing my changes to pyopenssl out I discovered it still happened without any of my new code. So… does pyopenssl not work with the new beta? I doubted it. Installed valgrind and run the test under that it showed somehow a shared library of crypto.so was getting loaded. dpkg -S showed that was part of the openssl package. Huh.

Puzzled about this for a while, but ldd /usr/bin/python showed that it’s was dynamically linked to openssl. Ahh… so pyopenssl was built with a static link to the new openssl, but python was also dynamically linking in the old one, there’s your segfaults. The two versions had overwrote each others symbols (unsure why it wasn’t all one or the other)

Then I worked on building openssl to be shared and be in /usr/local/lib so that they override the system ssl. I used stow as a way to put all the openssl files in their own place (/usr/local/stow/openssl-1.0.1-beta2) and then have stow symlink them to the right place.

Then I get this:

python: /usr/local/lib/libcrypto.so.1.0.0: no version information available (required by python)
python: /usr/local/lib/libssl.so.1.0.0: no version information available (required by python)

openssl out of the box doesn’t put versioning info on it’s .so’s, So I borrowed a patch from the debian package and applied it. Now openssl is failing to build some things.

And that’s where I need to stop for the night. :(

Here is a patch for pyopenssl that might add NPN to it, or it might cause your mouth to taste of metal for the rest of your life.

Project: txsdpy

A ShRUG this week someone mentioned a guy who was working a project every night and blogging about the progress he was making. He was up to 200+ days of this. I thought I’d try to do the same.

So first project: txspdy - a python twisted implementation of a SPDY server.

Why?

1. knowing http has helped in my day job and protocols are fun. XMPP hacking was a lot of fun.
2. new shiny
3. websockets over spdy sounds awesome. I’ve used websockets to build a app UI for a DirectConnect client I’ve written.

Why hasn’t anyone else done this yet?

There is a python prototype linked off the SDPY page, but it’s doesn’t use twisted. I will, no doubt, be reading over that code. The fun thing is that no one knows if SDPY will take off.

First problem

SPDY uses a new SSL extension to tell connecting clients what protocols the server can speak. OpenSSL has a beta version that implements this extension, but you can’t use it via pyopenssl. So I’ve checked that out and started hacking on the C code for it.

I wanted to test that I could get Chrome to talk SPDY to a test server before I went much further. First thing you need are some real ssl certs. I had some from StartCom. I used the openssl binary like this:

/usr/local/ssl/bin/openssl s_server -key server.key \
   -cert server.crt -accept 9000 -www -nextprotoneg spdy/2,http/1.1 -debug

My first attempts with a self signed cert resulted in Chrome connecting and stopping the ssl handshake 3 times then using http/1.1 even thought I had spdy listed too. Not sure what the purpose of that was.

From 802.11g to 802.11n

File Transfer speed:

802.11g: 98MB 2.1MB/s 00:47

802.11n: 98MB 4.7MB/s 00:21

Beware of ECS A740GM-M and 4+ GB of ram

My ZFS storage server needed more ram so that I could get the VM’s running on it a bit more elbow room. I got a pair of 4GB 240 Ram chips and they were a no go on the ECS A740GM-M motherboard I have. Seems that the BIOS screen needs to have the option to remap memory, which it does not. Too bad the manufactures page for it says it supports up to 16GB

Updating GCD Data

So you have loaded the Grand Comicbook Database into a local postgresql instance and wrote some code that makes use of the data… They just did a new data dump… Now how do you update your copy of the data?

Prep the data

Do the steps in “create mysql clean up script” and “dump data to tab separated value files” steps.

Now copy this python script:

#!/usr/bin/env python

"""
update gcd data that's prep'ed in /tmp/gcd_dump
"""

import os, glob
from pprint import pprint
import psycopg2, psycopg2.extras

table_names = [os.path.splitext(os.path.basename(fp))[0] for fp in glob.glob('/tmp/gcd_dump/*.txt')]

conn = psycopg2.connect("dbname='gcd' user='postgres'")
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)

def sql_logger(sql):
    print sql
    cur.execute(sql)

constraints = []
for ii in table_names:
    sql_logger("""
select t.constraint_name, t.table_name, t.constraint_type,
  c.table_name as c_table_name, c.column_name as c_column_name, k.column_name as k_column_name
from information_schema.table_constraints t,
  information_schema.constraint_column_usage c,
  information_schema.key_column_usage k

  where t.constraint_name = c.constraint_name
    and t.constraint_name = k.constraint_name
    and t.constraint_type = 'FOREIGN KEY'
    and c.table_name = '%s'
  """ % ii)
    for row in cur:
        constraints.append(dict(row))

sql_logger('begin')
for ii in constraints:
    sql_logger('alter table %s drop constraint %s;' % (ii['table_name'], ii['constraint_name'],))

for table_name in table_names:
    sql_logger("DELETE FROM %(table_name)s" % locals())
    sql_logger("COPY %(table_name)s FROM '/tmp/gcd_dump/%(table_name)s.txt'" % locals())

for ii in constraints:
    sql_logger("""
ALTER TABLE ONLY %(table_name)s
  ADD CONSTRAINT %(constraint_name)s
    FOREIGN KEY (%(k_column_name)s) REFERENCES %(c_table_name)s(%(c_column_name)s) DEFERRABLE INITIALLY DEFERRED;
""" % ii)

sql_logger('commit')

You’ll have to run this as the postgres user just as before. It records the FOREIGN KEY CONSTRAINT, drops them, deletes the old data, copies in the new, and recreates the constraints, all in one transaction! Eat that MySQL.