Shutup and Code

File Transfer speed:

802.11g: 98MB 2.1MB/s 00:47

802.11n: 98MB 4.7MB/s 00:21

May 26, 2010 (3 months, 1 week ago) | Tags: wifi | loading comment count...

My ZFS storage server needed more ram so that I could get the VM's running on it a bit more elbow room. I got a pair of 4GB 240 Ram chips and they were a no go on the ECS A740GM-M motherboard I have. Seems that the BIOS screen needs to have the option to remap memory, which it does not. Too bad the manufactures page for it says it supports up to 16GB

April 14, 2010 (4 months, 3 weeks ago) | loading comment count...

So you have loaded the Grand Comicbook Database into a local postgresql instance and wrote some code that makes use of the data... They just did a new data dump... Now how do you update your copy of the data?

Prep the data

Do the steps in "create mysql clean up script" and "dump data to tab separated value files" steps.

Now copy this python script:

#!/usr/bin/env python

"""
update gcd data that's prep'ed in /tmp/gcd_dump
"""

import os, glob
from pprint import pprint
import psycopg2, psycopg2.extras

table_names = [os.path.splitext(os.path.basename(fp))[0] for fp in glob.glob('/tmp/gcd_dump/*.txt')]

conn = psycopg2.connect("dbname='gcd' user='postgres'")
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)

def sql_logger(sql):
    print sql
    cur.execute(sql)

constraints = []
for ii in table_names:
    sql_logger("""
select t.constraint_name, t.table_name, t.constraint_type,
  c.table_name as c_table_name, c.column_name as c_column_name, k.column_name as k_column_name
from information_schema.table_constraints t,
  information_schema.constraint_column_usage c,
  information_schema.key_column_usage k

  where t.constraint_name = c.constraint_name
    and t.constraint_name = k.constraint_name
    and t.constraint_type = 'FOREIGN KEY'
    and c.table_name = '%s'
  """ % ii)
    for row in cur:
        constraints.append(dict(row))

sql_logger('begin')
for ii in constraints:
    sql_logger('alter table %s drop constraint %s;' % (ii['table_name'], ii['constraint_name'],))

for table_name in table_names:
    sql_logger("DELETE FROM %(table_name)s" % locals())
    sql_logger("COPY %(table_name)s FROM '/tmp/gcd_dump/%(table_name)s.txt'" % locals())

for ii in constraints:
    sql_logger("""
ALTER TABLE ONLY %(table_name)s 
  ADD CONSTRAINT %(constraint_name)s 
    FOREIGN KEY (%(k_column_name)s) REFERENCES %(c_table_name)s(%(c_column_name)s) DEFERRABLE INITIALLY DEFERRED;
""" % ii)

sql_logger('commit')

You'll have to run this as the postgres user just as before. It records the FOREIGN KEY CONSTRAINT, drops them, deletes the old data, copies in the new, and recreates the constraints, all in one transaction! Eat that MySQL.

Feb. 2, 2010 (7 months ago) | Tags: django,python | loading comment count...

I got tired of putting

import os, sys
sys.path.append(<django project parent dir>)
sys.path.append(<django project dir>)
os.environ['DJANGO_SETTINGS_MODULE']='<django project name>.settings'

at the top of all my scripts that do command line things with my django models. So I share with you 'django_loader.py'. Note the use of traceback to figure out what file is importing 'django_loader.py'.

""" 
Put this in your python path.  At the top of your script put 'import
django_loader'.  This will start with the directory your file is in and
search through it and it's parent directories until it finds a file named
'settings.py'.  It will then add that directory and it's parent to your
sys.path, and set DJANGO_SETTINGS env var.  
"""

import os, sys, traceback

class CouldNotFindSettings(StandardError):
    pass
def find_settings(current_dir):
    if current_dir == '/':
        raise CouldNotFindSettings
    if 'settings.py' in os.listdir(current_dir):
        return current_dir
    return find_settings(os.path.dirname(current_dir))
def load(filepath):
    django_project_dir = find_settings(os.path.dirname(filepath))
    django_project_name = os.path.basename(django_project_dir)

    sys.path.append(os.path.dirname(django_project_dir))
    sys.path.append(django_project_dir)
    os.environ['DJANGO_SETTINGS_MODULE']='%s.settings' % (django_project_name,)

current_filepath = os.path.normpath(os.path.join(os.getcwd(), traceback.extract_stack(limit=2)[0][0]))
load(current_filepath)

Jan. 21, 2010 (7 months, 2 weeks ago) | Tags: django,python | loading comment count...

XBox 360

• Dead Rising - I've been playing this off and on since 2008. This last weekend I made a huge amount of progress, but I might have to start over due to saving when I was almost out of time for a mission. Punishing difficulty, but it's more fun that way.
• Dragon Age - There's no way I'll finish before Mass Effect 2 comes out this month, and I like the Mass Effect story more. I suspect this one will be around for a while.
• Left 4 Dead 2 - It's intense, but difficult to get the right people for a good game at expert. Clearly I need a second TV and xbox to put in the family room.

D&D

Just finished a 3 year once a week campaign. My character destroyed the universe. Sorry about that guys.

Started a new campaign as a Binder/Bard based way too closely on Eddie Riggs from Brütal Legend. I know it's cheep to copy, but it's fun. Since the world he lives in looks more like the a heavy metal cover from this world his album covers will have folks in offices, staring at computers.

Jan. 11, 2010 (7 months, 3 weeks ago) | Tags: video-games | loading comment count...

I use Django and Postgres at home, because I use Rails and MySQL all day at work. Working in totally different solutions to the same problem keeps you fresh.

Jan. 11, 2010 (7 months, 3 weeks ago) | loading comment count...

To do this you need both mysql and postgresql running on a local computer. You probably want this to be a local workstation that you have superuser access to. We are going to use features in mysql and postgres that makes the database daemon read and write to local files.

We'll use django's schema format deal with the difference between postgresql and mysql. We'll use tab separated value (TSV) data files as the interchange format between databases. Mysql has a different idea of how to escape newlines and carriage returns than Postgresql so we'll use a quick and dirty python script to clean that up.

While this should work in many different OS's, I did this on a Ubuntu, so the details might be a bit different.

Let's start with the most recent data dump from the Grand Comicbook Database

load data into mysql

mysqladmin -uroot create gcd
mysql -uroot gcd < pub_dec21_schema_innodb.sql
unzip pub_dec21_data.zip
mysql -uroot gcd < pub_dec21_data.sql

create django project

django-admin.py startproject grandcomicdb
cd grandcomicdb
chmod +x manage.py
./manage.py startapp gcd
# edit settings.py to add gcd to INSTALLED_APPS
# edit settings.py to set up connection to mysql
./manage.py inspectdb > gcd/models.py
# edit gcd/models.py to make the fk quoted, and add relative_name's

create mysql clean up script

cat >> fix_mysql_tsv.py << EOF
#!/usr/bin/env python

# this will not work for very big files.

import sys
ff = open(sys.argv[1], 'r').read()
ff = ff.replace('\r', '\\r')
ff = ff.replace('\\\n', '\\n')

open(sys.argv[1], 'w').write(ff)
EOF
chmod +x fix_mysql_tsv.py

dump data to tab separated value files

mkdir /tmp/gcd_dump
chmod 777 /tmp/gcd_dump
mysqldump -uroot -t --tab /tmp/gcd_dump gcd
find /tmp/gcd_dump -type f -exec ~/web/grandcomicsdb/fix_mysql_tsv.py \{\} \;

create postgres database with schema derived from the mysql database

sudo -s -u postgres
createuser gcd --pwprompt --no-createrole --no-createdb
createdb gcd -O gcd
exit
# edit settings.py to set up connection to postgresql
./manage syncdb

create postgres database and load data

sudo -s -u postgres
psql
BEGIN;
COPY gcd_language FROM '/tmp/gcd_dump/gcd_language.txt';
COPY gcd_country FROM '/tmp/gcd_dump/gcd_country.txt';
COPY gcd_brand FROM '/tmp/gcd_dump/gcd_brand.txt';
COPY gcd_publisher FROM '/tmp/gcd_dump/gcd_publisher.txt';
COPY gcd_indicia_publisher FROM '/tmp/gcd_dump/gcd_indicia_publisher.txt';
COPY gcd_story_type FROM '/tmp/gcd_dump/gcd_story_type.txt';
COPY gcd_series FROM '/tmp/gcd_dump/gcd_series.txt';
COPY gcd_issue FROM '/tmp/gcd_dump/gcd_issue.txt';
COPY gcd_story FROM '/tmp/gcd_dump/gcd_story.txt';
COMMIT;

Jan. 11, 2010 (7 months, 3 weeks ago) | Tags: django,python | loading comment count...

I couldn't find the simplest example of using Ruby's SVN bindings. Here's something simple, get the info on some file in a local working directory.

require 'svn/client'

ctx = Svn::Client::Context.new
ctx.add_simple_provider
ctx.info('some file in your svn working dir') do |path,info|
  p path
  p info.last_changed_rev
end

This page is also useful.

Sept. 25, 2008 (1 year, 11 months ago) | Tags: ruby | loading comment count...

This should give you an idea of what it’s like to work at Rosetta Stone… the CEO just emailed this to everyone:

Sept. 20, 2008 (1 year, 11 months ago) | loading comment count...

Like you care…

XBox 360

• Dead Rising (love it)
• GTA IV
• Mass Effect
• Beautiful Katamari
• Half Life 2 (replaying for the achievements)
• Castle Crashers (need to try online coop)
• Braid (hurting my brain)

PS2

• Champions of Norrath 2 (waiting to visit Jason and hack our way thru)
• God of War (suck in that room of blades on a grid that needs timing that I can’t do)

Wii

• Mario Galaxy

Sept. 12, 2008 (1 year, 11 months ago) | Tags: video-games | loading comment count...

    xml = generate_xml
    require 'xml'
    Tempfile.open(self.class.to_s) do |tmp|
      tmp.write(xml)
      tmp.close
      document = XML::Document.file(tmp.path)
      schema_doc = XML::Document.file("some.xsd")
      schema = XML::Schema.document(schema_doc)
      assert document.validate(schema), "the xml isn't valid.  look above for error."
    end

Sept. 6, 2008 (1 year, 12 months ago) | loading comment count...

gem install s3sync

Has the right idea about how to store files, but the lib it uses does not abstract things enough. Does not use http keep alive to avoid the slow startup of tcp. Does not have an easy way to iterate over all keys in a bucket, but this helps to do this:

  def each_object(bucket_name)
    next_marker = nil
    while true do
      response = CONN.list_bucket(bucket_name, {'marker' => next_marker, 'max-keys' => 10})
      response.entries.each do |s3obj|  
        yield s3obj
      end
      break unless response.properties.is_truncated
      next_marker = response.entries.last.key
    end
  end

gem install aws-s3

Again no built in way to iterate over all keys. Has problems with '/' at the start of file names.

gem install right_aws

Uses the same http connection, can iterate over all keys with a single method (though, it makes a full array of all the keys rather than allowing you to supply a block). Here's my thumbnailer:

  require 'rubygems'
  require 'right_aws'    
  require 'RMagick'
  require 'pp'

  s3 = RightAws::S3.new(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)

  picture_bucket = s3.bucket('OurPictures')
  thumbnail_bucket = s3.bucket('OurPicturesThumbnails')

  picture_bucket.keys.each do |key|
    thumbnail_key = RightAws::S3::Key.create(thumbnail_bucket, key.name)
    next if key.name !~ /.jpg$/i
    next if thumbnail_key.exists?
    image = Magick::Image.from_blob(key.data).first
    image.change_geometry!('256x256>') do |cols, rows, img|
      img.thumbnail!(cols, rows)
    end
    thumbnail_key.put(image.to_blob, 'private')
    p thumbnail_key.full_name
    image = nil
    GC.start
  end

May 31, 2008 (2 years, 3 months ago) | Tags: ruby | loading comment count...

(mostly randy here again. I was putting up my first ec2 instance. :)

Saturday Keynotes

• funny video from the RailsEnvy guys about testing

DHH introducing Jeremy Kemper

• Jeremy has done a buttload of work lately in the Rails core
• dhh on a coding vacation?
• some people think about one part of rails Jeremy thinks about the whole thing

Jeremy’s talk about Rails 2… where it’s going, etc

• “it’s all about resources”
• “we shed a lot of fat” – split things off into plugins
• “we gained speed” – “i’m not concerned about rails being super super quick” — huh???
• 1600 patches… it’s hard to read all that code! so we embraced git and lighthouse instead of svn and trac
• Rails 2.1
  • refactoring
  • documentation
  • thinner + faster
  • (he looks exhausted… too much free five runs beer and cheese pies)
  • he wants rails to look pretty.
  • use rubyprof
  • merging migrations
  • making timezones fitter, happier, more productive
  • migrations now have change_table block for modifying tables (just like create_table)
  • gem dependencies
  • improved memcache… making it a first-class citizen in rails. memcache-client now bundled with rails
  • much of what he’s covering Myers read in this tutorial on rails 2.1
  • dirty – AR now knows when an attribute has changed
    • message.body_changed?
    • message.body_was
    • message.changed?
    • enables partial updates
  • smarter :include — it’s not as eager to join on the first query. this seems like it will be slower, but benchmarking supposedly proves that querying the join table on an as-needed basis is faster
  • named_scope
    • so you can say user.messages.recent instead of user.recent_messages by saying named_scope :recent, :order -> 'created_at desc’ in the Message model
  • Message.scoped(:limit => 10)
  • jruby -S jetty_rails (run rails on jruby)
  • rbx script/server (run rails on rubinius)
  • rails now runs on ruby 1.9
• rails 2.1 out today (a gem update at 10:14am didn’t do anything tho… it will be later… so not a Steve Jobs “and you can buy it right now”)

May 31, 2008 (2 years, 3 months ago) | Tags: railsconf2008 | loading comment count...

DHH keynote

• surplus of productivity
• company’s needs that special in the problems we encounter
• We ceded flexibility
• “people like choices a lot more than they like to choose” – dhh, just now
• why is this not in the framework, someone make this choice for me
• we decided tech matters
• “great people rarely fail because of poor technology” – but come on, do you want to just “not fail”?
• we cared about us
• “ruby is designed to make programmers happy” – matz
• the surplus will not last forever
  • why? the mainstream will copy rails (dhh doesn’t think so)
  • dramatic alternative arrives (dhh doesn’t think so)
  • rails becomes the mainstream
• business as usual – which is blowing your surplus, running at 110% all the time
• another day, another fire
• Lost in the mechanics – when running at 110% means you can’t see anything else
• fatigued, disinterested, passionless
• it’s just a job – most depressing statement
• one place to invest that will pay back: you
• 1:10 programmer productivity
• no one is born a rock star programmer, you become it
• recharge tangentially – or some other word, do something else besides sit in front of a computer all day: making spoons, play the banjo, fly a plane
• can’t just focus on one muscle
• speaking about taking people to the next level: sleep more (applause)
• stop to read paper
• suggested reading
  • my job went to india (and all i got was this lousy book)
  • implementation patterns
  • innovator’s dilemma
  • tufte’s envisioning information
• all this helps you judge what is valuable or not
• is this lack of a seam on the iphone what matters.
• program less
• when you have only 10 hours to program a week, you know what matters
• sometimes good to start a project from scratch
• share: you benefit from the sharing
• “the purpose of playing this game well is to be able to get the best position of the next game” – alistair cockburn, 1999
• 4 day work week means more focus
• summary: this surplus is not going to last forever don’t blow it all on hookers and fur coats

May 31, 2008 (2 years, 3 months ago) | Tags: railsconf2008 | loading comment count...

Dialogue Concerning the Two Chief Modeling Systems

• it’s a Play

(again you should read the slides)

• choosing the right name will make the dev think about this model and give it the right property
• Jim keeps saying we’re skipping layers
• using story cards
• jammed up over reoccurring events
• objects = data + behavior, so you can’t just talk about the data (rows/tables), you must look at behavior
• pull out the Class, Responsibilities, and Collaborator cards
• card only useful for organizing your thoughts not need to fill them all out
• “temporal expressions”
• CJ Date: apparently wrote a lot about data modeling
• Code Smell in Refactoring by Chad Fowler

Flexible Scaling: How to Handle 1 Billion Pageviews – TJ

• WoW: you see all kinds of human behaviors: Mafia, Philantropis
• Building games on Facebook
• He’s the author of Warbook: Rise of the Infernals
• w/i a week he had to fix stuff
• w/i 3 weeks had to rewrite
• started with firebug’s net tab
• look at your logs: pl_analyze
• iostat

• you need tools, but you also need strategies
• don’t need it? ditch it.
• slowing it down? simplify it.
• logging it? stop
• selecting it? cache it.
• memcache
• put sessions in cache
• no-select design
  • use memcache
  • cache_fu already does this
• using ec2
• 1 db box
• 1 memcache box
• 1 static file
• X mongrel boxes
• The Hard Part
  • Scale Everything Else
  • Scale your deploymnet
    • use capistrano
  • Scaling your support
  • community management
    • give them updates every day
• server cost $2000 a month
• you need money
• warbook makes $100,000/month
• 1.5 million users
• 16 million page views

Q&A

• remove transactional saves
• save per fields
• how to solve the persistence problem on ec2? db not on amazon
• which facebook lib? started with rfacebook, facebooker, bebo

Slides

May 30, 2008 (2 years, 3 months ago) | Tags: railsconf2008 | loading comment count...