File Transfer speed:
802.11g: 98MB 2.1MB/s 00:47
802.11n: 98MB 4.7MB/s 00:21
May 26, 2010 (3 months, 1 week ago) | Tags: wifi | loading comment count...
File Transfer speed:
802.11g: 98MB 2.1MB/s 00:47
802.11n: 98MB 4.7MB/s 00:21
May 26, 2010 (3 months, 1 week ago) | Tags: wifi | loading comment count...
My ZFS storage server needed more ram so that I could get the VM's running on it a bit more elbow room. I got a pair of 4GB 240 Ram chips and they were a no go on the ECS A740GM-M motherboard I have. Seems that the BIOS screen needs to have the option to remap memory, which it does not. Too bad the manufactures page for it says it supports up to 16GB
April 14, 2010 (4 months, 3 weeks ago) | loading comment count...
So you have loaded the Grand Comicbook Database into a local postgresql instance and wrote some code that makes use of the data... They just did a new data dump... Now how do you update your copy of the data?
Do the steps in "create mysql clean up script" and "dump data to tab separated value files" steps.
Now copy this python script:
#!/usr/bin/env python
"""
update gcd data that's prep'ed in /tmp/gcd_dump
"""
import os, glob
from pprint import pprint
import psycopg2, psycopg2.extras
table_names = [os.path.splitext(os.path.basename(fp))[0] for fp in glob.glob('/tmp/gcd_dump/*.txt')]
conn = psycopg2.connect("dbname='gcd' user='postgres'")
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
def sql_logger(sql):
print sql
cur.execute(sql)
constraints = []
for ii in table_names:
sql_logger("""
select t.constraint_name, t.table_name, t.constraint_type,
c.table_name as c_table_name, c.column_name as c_column_name, k.column_name as k_column_name
from information_schema.table_constraints t,
information_schema.constraint_column_usage c,
information_schema.key_column_usage k
where t.constraint_name = c.constraint_name
and t.constraint_name = k.constraint_name
and t.constraint_type = 'FOREIGN KEY'
and c.table_name = '%s'
""" % ii)
for row in cur:
constraints.append(dict(row))
sql_logger('begin')
for ii in constraints:
sql_logger('alter table %s drop constraint %s;' % (ii['table_name'], ii['constraint_name'],))
for table_name in table_names:
sql_logger("DELETE FROM %(table_name)s" % locals())
sql_logger("COPY %(table_name)s FROM '/tmp/gcd_dump/%(table_name)s.txt'" % locals())
for ii in constraints:
sql_logger("""
ALTER TABLE ONLY %(table_name)s
ADD CONSTRAINT %(constraint_name)s
FOREIGN KEY (%(k_column_name)s) REFERENCES %(c_table_name)s(%(c_column_name)s) DEFERRABLE INITIALLY DEFERRED;
""" % ii)
sql_logger('commit')
You'll have to run this as the postgres user just as before. It records the FOREIGN KEY CONSTRAINT, drops them, deletes the old data, copies in the new, and recreates the constraints, all in one transaction! Eat that MySQL.
Feb. 2, 2010 (7 months ago) | Tags: django,python | loading comment count...
I got tired of putting
import os, sys
sys.path.append(<django project parent dir>)
sys.path.append(<django project dir>)
os.environ['DJANGO_SETTINGS_MODULE']='<django project name>.settings'
at the top of all my scripts that do command line things with my django models. So I share with you 'django_loader.py'. Note the use of traceback to figure out what file is importing 'django_loader.py'.
"""
Put this in your python path. At the top of your script put 'import
django_loader'. This will start with the directory your file is in and
search through it and it's parent directories until it finds a file named
'settings.py'. It will then add that directory and it's parent to your
sys.path, and set DJANGO_SETTINGS env var.
"""
import os, sys, traceback
class CouldNotFindSettings(StandardError):
pass
def find_settings(current_dir):
if current_dir == '/':
raise CouldNotFindSettings
if 'settings.py' in os.listdir(current_dir):
return current_dir
return find_settings(os.path.dirname(current_dir))
def load(filepath):
django_project_dir = find_settings(os.path.dirname(filepath))
django_project_name = os.path.basename(django_project_dir)
sys.path.append(os.path.dirname(django_project_dir))
sys.path.append(django_project_dir)
os.environ['DJANGO_SETTINGS_MODULE']='%s.settings' % (django_project_name,)
current_filepath = os.path.normpath(os.path.join(os.getcwd(), traceback.extract_stack(limit=2)[0][0]))
load(current_filepath)
Jan. 21, 2010 (7 months, 2 weeks ago) | Tags: django,python | loading comment count...
Just finished a 3 year once a week campaign. My character destroyed the universe. Sorry about that guys.
Started a new campaign as a Binder/Bard based way too closely on Eddie Riggs from BrĂ¼tal Legend. I know it's cheep to copy, but it's fun. Since the world he lives in looks more like the a heavy metal cover from this world his album covers will have folks in offices, staring at computers.
Jan. 11, 2010 (7 months, 3 weeks ago) | Tags: video-games | loading comment count...
I use Django and Postgres at home, because I use Rails and MySQL all day at work. Working in totally different solutions to the same problem keeps you fresh.
Jan. 11, 2010 (7 months, 3 weeks ago) | loading comment count...
To do this you need both mysql and postgresql running on a local computer. You probably want this to be a local workstation that you have superuser access to. We are going to use features in mysql and postgres that makes the database daemon read and write to local files.
We'll use django's schema format deal with the difference between postgresql and mysql. We'll use tab separated value (TSV) data files as the interchange format between databases. Mysql has a different idea of how to escape newlines and carriage returns than Postgresql so we'll use a quick and dirty python script to clean that up.
While this should work in many different OS's, I did this on a Ubuntu, so the details might be a bit different.
Let's start with the most recent data dump from the Grand Comicbook Database
mysqladmin -uroot create gcd
mysql -uroot gcd < pub_dec21_schema_innodb.sql
unzip pub_dec21_data.zip
mysql -uroot gcd < pub_dec21_data.sql
django-admin.py startproject grandcomicdb
cd grandcomicdb
chmod +x manage.py
./manage.py startapp gcd
# edit settings.py to add gcd to INSTALLED_APPS
# edit settings.py to set up connection to mysql
./manage.py inspectdb > gcd/models.py
# edit gcd/models.py to make the fk quoted, and add relative_name's
cat >> fix_mysql_tsv.py << EOF
#!/usr/bin/env python
# this will not work for very big files.
import sys
ff = open(sys.argv[1], 'r').read()
ff = ff.replace('\r', '\\r')
ff = ff.replace('\\\n', '\\n')
open(sys.argv[1], 'w').write(ff)
EOF
chmod +x fix_mysql_tsv.py
mkdir /tmp/gcd_dump
chmod 777 /tmp/gcd_dump
mysqldump -uroot -t --tab /tmp/gcd_dump gcd
find /tmp/gcd_dump -type f -exec ~/web/grandcomicsdb/fix_mysql_tsv.py \{\} \;
sudo -s -u postgres
createuser gcd --pwprompt --no-createrole --no-createdb
createdb gcd -O gcd
exit
# edit settings.py to set up connection to postgresql
./manage syncdb
sudo -s -u postgres
psql
BEGIN;
COPY gcd_language FROM '/tmp/gcd_dump/gcd_language.txt';
COPY gcd_country FROM '/tmp/gcd_dump/gcd_country.txt';
COPY gcd_brand FROM '/tmp/gcd_dump/gcd_brand.txt';
COPY gcd_publisher FROM '/tmp/gcd_dump/gcd_publisher.txt';
COPY gcd_indicia_publisher FROM '/tmp/gcd_dump/gcd_indicia_publisher.txt';
COPY gcd_story_type FROM '/tmp/gcd_dump/gcd_story_type.txt';
COPY gcd_series FROM '/tmp/gcd_dump/gcd_series.txt';
COPY gcd_issue FROM '/tmp/gcd_dump/gcd_issue.txt';
COPY gcd_story FROM '/tmp/gcd_dump/gcd_story.txt';
COMMIT;
Jan. 11, 2010 (7 months, 3 weeks ago) | Tags: django,python | loading comment count...
I couldn't find the simplest example of using Ruby's SVN bindings. Here's something simple, get the info on some file in a local working directory.
require 'svn/client'
ctx = Svn::Client::Context.new
ctx.add_simple_provider
ctx.info('some file in your svn working dir') do |path,info|
p path
p info.last_changed_rev
end
This page is also useful.
Sept. 25, 2008 (1 year, 11 months ago) | Tags: ruby | loading comment count...
This should give you an idea of what it’s like to work at Rosetta Stone… the CEO just emailed this to everyone:
Sept. 20, 2008 (1 year, 11 months ago) | loading comment count...
Like you care…
Sept. 12, 2008 (1 year, 11 months ago) | Tags: video-games | loading comment count...
xml = generate_xml
require 'xml'
Tempfile.open(self.class.to_s) do |tmp|
tmp.write(xml)
tmp.close
document = XML::Document.file(tmp.path)
schema_doc = XML::Document.file("some.xsd")
schema = XML::Schema.document(schema_doc)
assert document.validate(schema), "the xml isn't valid. look above for error."
end
Sept. 6, 2008 (1 year, 12 months ago) | loading comment count...
Has the right idea about how to store files, but the lib it uses does not abstract things enough. Does not use http keep alive to avoid the slow startup of tcp. Does not have an easy way to iterate over all keys in a bucket, but this helps to do this:
def each_object(bucket_name)
next_marker = nil
while true do
response = CONN.list_bucket(bucket_name, {'marker' => next_marker, 'max-keys' => 10})
response.entries.each do |s3obj|
yield s3obj
end
break unless response.properties.is_truncated
next_marker = response.entries.last.key
end
end
Again no built in way to iterate over all keys. Has problems with '/' at the start of file names.
Uses the same http connection, can iterate over all keys with a single method (though, it makes a full array of all the keys rather than allowing you to supply a block). Here's my thumbnailer:
require 'rubygems'
require 'right_aws'
require 'RMagick'
require 'pp'
s3 = RightAws::S3.new(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
picture_bucket = s3.bucket('OurPictures')
thumbnail_bucket = s3.bucket('OurPicturesThumbnails')
picture_bucket.keys.each do |key|
thumbnail_key = RightAws::S3::Key.create(thumbnail_bucket, key.name)
next if key.name !~ /.jpg$/i
next if thumbnail_key.exists?
image = Magick::Image.from_blob(key.data).first
image.change_geometry!('256x256>') do |cols, rows, img|
img.thumbnail!(cols, rows)
end
thumbnail_key.put(image.to_blob, 'private')
p thumbnail_key.full_name
image = nil
GC.start
end
May 31, 2008 (2 years, 3 months ago) | Tags: ruby | loading comment count...
(mostly randy here again. I was putting up my first ec2 instance. :)
May 31, 2008 (2 years, 3 months ago) | Tags: railsconf2008 | loading comment count...
May 31, 2008 (2 years, 3 months ago) | Tags: railsconf2008 | loading comment count...
(again you should read the slides)
May 30, 2008 (2 years, 3 months ago) | Tags: railsconf2008 | loading comment count...