txspdy: Day 4: frames and streams oh my

Last night was light on hacking time. Did a lot of reading of the twisted.web code. What I would like to end up way to create a duel https/spdy exposing the same Resource instance tree. Unfortunately it seems like the Site class is a subclass of a HTTPFactory where it would be better for this if it had a ref to that factory.

I used some code from nbhttp to parse the frames and frame headers for a single request. Next up is creating the headers and response frame. Then we’ll have enough for a “hello world”.

txspdy: Day 3: pyopenssl done... now to python code

I don’t know where to start talking about the problems with building your own copy of Openssl. Ubuntu has added a few patches, one of which adds symbol versioning to the openssl shared libraries. Sounds great, but it means you can’t build a new version of openssl from upstream and install it system wide without problems, like say your openssh server dies because of it and you can’t login. I have a monitoring system, icinga, that noticed this and sent emails/txt msgs about it so I fixed the problem before finding myself locked out of the system.

Once that was settled I got the three new methods added to pyopenssl and made a test as well. I’ve sent it upstream to get merged. You can see it at LaunchPad.

Now I’m writing a twisted protocol that will parse SPDY frames. The frames build to multiple streams, and I’m not sure how to deal with those. Maybe there is some other multiplexing protocol like this already in twisted.

txspdy: Day 2

Fighting with C. You hate loving it. You love hating it. But like water, it’s good to know how to swim in it because 3/4 of the computing infrastructure is built on it.

I needed to build the openssl-1.0.1-beta2, then make a mod of pyopenssl to add in the NPN feature. Both of these went fine, and I got to a point that I had a clean compile. Trying to run a unit test showed me that something was segfaulting. Backing my changes to pyopenssl out I discovered it still happened without any of my new code. So… does pyopenssl not work with the new beta? I doubted it. Installed valgrind and run the test under that it showed somehow a shared library of crypto.so was getting loaded. dpkg -S showed that was part of the openssl package. Huh.

Puzzled about this for a while, but ldd /usr/bin/python showed that it’s was dynamically linked to openssl. Ahh… so pyopenssl was built with a static link to the new openssl, but python was also dynamically linking in the old one, there’s your segfaults. The two versions had overwrote each others symbols (unsure why it wasn’t all one or the other)

Then I worked on building openssl to be shared and be in /usr/local/lib so that they override the system ssl. I used stow as a way to put all the openssl files in their own place (/usr/local/stow/openssl-1.0.1-beta2) and then have stow symlink them to the right place.

Then I get this:

python: /usr/local/lib/libcrypto.so.1.0.0: no version information available (required by python)
python: /usr/local/lib/libssl.so.1.0.0: no version information available (required by python)

openssl out of the box doesn’t put versioning info on it’s .so’s, So I borrowed a patch from the debian package and applied it. Now openssl is failing to build some things.

And that’s where I need to stop for the night. :(

Here is a patch for pyopenssl that might add NPN to it, or it might cause your mouth to taste of metal for the rest of your life.

Project: txsdpy

A ShRUG this week someone mentioned a guy who was working a project every night and blogging about the progress he was making. He was up to 200+ days of this. I thought I’d try to do the same.

So first project: txspdy - a python twisted implementation of a SPDY server.

Why?

1. knowing http has helped in my day job and protocols are fun. XMPP hacking was a lot of fun.
2. new shiny
3. websockets over spdy sounds awesome. I’ve used websockets to build a app UI for a DirectConnect client I’ve written.

Why hasn’t anyone else done this yet?

There is a python prototype linked off the SDPY page, but it’s doesn’t use twisted. I will, no doubt, be reading over that code. The fun thing is that no one knows if SDPY will take off.

First problem

SPDY uses a new SSL extension to tell connecting clients what protocols the server can speak. OpenSSL has a beta version that implements this extension, but you can’t use it via pyopenssl. So I’ve checked that out and started hacking on the C code for it.

I wanted to test that I could get Chrome to talk SPDY to a test server before I went much further. First thing you need are some real ssl certs. I had some from StartCom. I used the openssl binary like this:

/usr/local/ssl/bin/openssl s_server -key server.key \
   -cert server.crt -accept 9000 -www -nextprotoneg spdy/2,http/1.1 -debug

My first attempts with a self signed cert resulted in Chrome connecting and stopping the ssl handshake 3 times then using http/1.1 even thought I had spdy listed too. Not sure what the purpose of that was.

From 802.11g to 802.11n

File Transfer speed:

802.11g: 98MB 2.1MB/s 00:47

802.11n: 98MB 4.7MB/s 00:21