Higher level protocols for websockets

I ran across WAMP recently. When I built my DC++ client who's primary GUI is a jquery/jquery ui web app connected to the twisted python backend via websocket I had to create something just like this. Later reading over the JSON RPC 2.0 spec I realized you could just use their format and leverage json rpc libs already written.

Small note: you might want to remove the "jsonrpc" key from the request and response to save some bytes.

There is also this: Cutting Down Bandwidth with JSON Alternatives

permalink | 19 Apr 2013 | tags: ideas

Google Glass App Idea: Am I talking to my kids enough?

The Power of Talking to Your Baby

If you haven’t heard of Hart and Risley’s work, you are not alone — and you may be wondering why. These findings should have created a policy whirlwind: Here was a revolutionary way to reduce inequities in school achievement that seemed actually possible. How hard could it be to persuade poor parents to talk to their children more?

Very hard, it turned out — because there was no practical way to measure how much parents talk. Each hour of recording took many hours to transcribe and classify: to count the words uttered near a child and attribute them to a parent, the main child, a sibling, someone else or a TV. The cost was prohibitive.

With Google Glass you could monitor everything you say, you could track if your kids are around you.

permalink | 15 Apr 2013 | tags: ideas

txspdy: Day 4: frames and streams oh my

Last night was light on hacking time. Did a lot of reading of the twisted.web code. What I would like to end up way to create a duel https/spdy exposing the same Resource instance tree. Unfortunately it seems like the Site class is a subclass of a HTTPFactory where it would be better for this if it had a ref to that factory.

I used some code from nbhttp to parse the frames and frame headers for a single request. Next up is creating the headers and response frame. Then we'll have enough for a "hello world".

permalink | 11 Feb 2012 | tags: txspdy

txspdy: Day 3: pyopenssl done... now to python code

I don't know where to start talking about the problems with building your own copy of Openssl. Ubuntu has added a few patches, one of which adds symbol versioning to the openssl shared libraries. Sounds great, but it means you can't build a new version of openssl from upstream and install it system wide without problems, like say your openssh server dies because of it and you can't login. I have a monitoring system, icinga, that noticed this and sent emails/txt msgs about it so I fixed the problem before finding myself locked out of the system.

Once that was settled I got the three new methods added to pyopenssl and made a test as well. I've sent it upstream to get merged. You can see it (at LaunchPad)[https://code.launchpad.net/~myers-1/pyopenssl/npn].

Now I'm writing a twisted protocol that will parse SPDY frames. The frames build to multiple streams, and I'm not sure how to deal with those. Maybe there is some other multiplexing protocol like this already in twisted.

permalink | 09 Feb 2012 | tags: txspdy

txspdy: Day 2

Fighting with C. You hate loving it. You love hating it. But like water, it's good to know how to swim in it because 3/4 of the computing infrastructure is built on it.

I needed to build the openssl-1.0.1-beta2, then make a mod of pyopenssl to add in the NPN feature. Both of these went fine, and I got to a point that I had a clean compile. Trying to run a unit test showed me that something was segfaulting. Backing my changes to pyopenssl out I discovered it still happened without any of my new code. So... does pyopenssl not work with the new beta? I doubted it. Installed valgrind and run the test under that it showed somehow a shared library of crypto.so was getting loaded. dpkg -S showed that was part of the openssl package. Huh.

Puzzled about this for a while, but ldd /usr/bin/python showed that it's was dynamically linked to openssl. Ahh... so pyopenssl was built with a static link to the new openssl, but python was also dynamically linking in the old one, there's your segfaults. The two versions had overwrote each others symbols (unsure why it wasn't all one or the other)

Then I worked on building openssl to be shared and be in /usr/local/lib so that they override the system ssl. I used stow as a way to put all the openssl files in their own place (/usr/local/stow/openssl-1.0.1-beta2) and then have stow symlink them to the right place.

Then I get this:

python: /usr/local/lib/libcrypto.so.1.0.0: no version information available (required by python)
python: /usr/local/lib/libssl.so.1.0.0: no version information available (required by python)

openssl out of the box doesn't put versioning info on it's .so's, So I borrowed a patch from the debian package and applied it. Now openssl is failing to build some things.

And that's where I need to stop for the night. :(

Here is a patch for pyopenssl that might add NPN to it, or it might cause your mouth to taste of metal for the rest of your life.

permalink | 08 Feb 2012 | tags: python, txspdy

Project: txsdpy

A ShRUG this week someone mentioned a guy who was working a project every night and blogging about the progress he was making. He was up to 200+ days of this. I thought I'd try to do the same.

So first project: txspdy - a python twisted implementation of a SPDY server.

Why?

  1. knowing http has helped in my day job and protocols are fun. XMPP hacking was a lot of fun.
  2. new shiny
  3. websockets over spdy sounds awesome. I've used websockets to build a app UI for a DirectConnect client I've written.

Why hasn't anyone else done this yet?

There is a python prototype linked off the SDPY page, but it's doesn't use twisted. I will, no doubt, be reading over that code. The fun thing is that no one knows if SDPY will take off.

First problem:

SPDY uses a new SSL extension to tell connecting clients what protocols the server can speak. OpenSSL has a beta version that implements this extension, but you can't use it via pyopenssl. So I've checked that out and started hacking on the C code for it.

I wanted to test that I could get Chrome to talk SPDY to a test server before I went much further. First thing you need are some real ssl certs. I had some from StartCom. I used the openssl binary like this:

/usr/local/ssl/bin/openssl s_server -key server.key \
   -cert server.crt -accept 9000 -www -nextprotoneg spdy/2,http/1.1 -debug

My first attempts with a self signed cert resulted in Chrome connecting and stopping the ssl handshake 3 times then using http/1.1 even thought I had spdy listed too. Not sure what the purpose of that was.

permalink | 08 Feb 2012 | tags: txspdy

From 802.11g to 802.11n

File Transfer speed:

802.11g: 98MB 2.1MB/s 00:47

802.11n: 98MB 4.7MB/s 00:21

permalink | 26 May 2010 | tags: wifi

Beware of ECS A740GM-M and 4+ GB of ram

My ZFS storage server needed more ram so that I could get the VM's running on it a bit more elbow room. I got a pair of 4GB 240 Ram chips and they were a no go on the ECS A740GM-M motherboard I have. Seems that the BIOS screen needs to have the option to remap memory, which it does not. Too bad the manufactures page for it says it supports up to 16GB

permalink | 14 Apr 2010 | tags:

Updating GCD Data

So you have loaded the Grand Comicbook Database into a local postgresql instance and wrote some code that makes use of the data... They just did a new data dump... Now how do you update your copy of the data?

Prep the data

Do the steps in "create mysql clean up script" and "dump data to tab separated value files" steps.

Now copy this python script:

#!/usr/bin/env python

"""
update gcd data that's prep'ed in /tmp/gcd_dump
"""

import os, glob
from pprint import pprint
import psycopg2, psycopg2.extras

table_names = [os.path.splitext(os.path.basename(fp))[0] for fp in glob.glob('/tmp/gcd_dump/*.txt')]

conn = psycopg2.connect("dbname='gcd' user='postgres'")
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)

def sql_logger(sql):
    print sql
    cur.execute(sql)

constraints = []
for ii in table_names:
    sql_logger("""
select t.constraint_name, t.table_name, t.constraint_type,
  c.table_name as c_table_name, c.column_name as c_column_name, k.column_name as k_column_name
from information_schema.table_constraints t,
  information_schema.constraint_column_usage c,
  information_schema.key_column_usage k

  where t.constraint_name = c.constraint_name
    and t.constraint_name = k.constraint_name
    and t.constraint_type = 'FOREIGN KEY'
    and c.table_name = '%s'
  """ % ii)
    for row in cur:
        constraints.append(dict(row))

sql_logger('begin')
for ii in constraints:
    sql_logger('alter table %s drop constraint %s;' % (ii['table_name'], ii['constraint_name'],))

for table_name in table_names:
    sql_logger("DELETE FROM %(table_name)s" % locals())
    sql_logger("COPY %(table_name)s FROM '/tmp/gcd_dump/%(table_name)s.txt'" % locals())

for ii in constraints:
    sql_logger("""
ALTER TABLE ONLY %(table_name)s 
  ADD CONSTRAINT %(constraint_name)s 
    FOREIGN KEY (%(k_column_name)s) REFERENCES %(c_table_name)s(%(c_column_name)s) DEFERRABLE INITIALLY DEFERRED;
""" % ii)

sql_logger('commit')  

You'll have to run this as the postgres user just as before. It records the FOREIGN KEY CONSTRAINT, drops them, deletes the old data, copies in the new, and recreates the constraints, all in one transaction! Eat that MySQL.

permalink | 02 Feb 2010 | tags: django, python

django_loader.py

I got tired of putting

import os, sys
sys.path.append(<django project parent dir>)
sys.path.append(<django project dir>)
os.environ['DJANGO_SETTINGS_MODULE']='<django project name>.settings'

at the top of all my scripts that do command line things with my django models. So I share with you 'djangoloader.py'. Note the use of traceback to figure out what file is importing 'djangoloader.py'.

""" 
Put this in your python path.  At the top of your script put 'import
django_loader'.  This will start with the directory your file is in and
search through it and it's parent directories until it finds a file named
'settings.py'.  It will then add that directory and it's parent to your
sys.path, and set DJANGO_SETTINGS env var.  
"""

import os, sys, traceback

class CouldNotFindSettings(StandardError):
    pass
def find_settings(current_dir):
    if current_dir == '/':
        raise CouldNotFindSettings
    if 'settings.py' in os.listdir(current_dir):
        return current_dir
    return find_settings(os.path.dirname(current_dir))
def load(filepath):
    django_project_dir = find_settings(os.path.dirname(filepath))
    django_project_name = os.path.basename(django_project_dir)

    sys.path.append(os.path.dirname(django_project_dir))
    sys.path.append(django_project_dir)
    os.environ['DJANGO_SETTINGS_MODULE']='%s.settings' % (django_project_name,)

current_filepath = os.path.normpath(os.path.join(os.getcwd(), traceback.extract_stack(limit=2)[0][0]))
load(current_filepath)

permalink | 21 Jan 2010 | tags: django, python