akalias

Cartesian

2011-01-04T19:33:00.000-08:00

I see your hacky method overloading ...

Luckily with Python you can implement your own hacky little overloads.

from itertools import product

def argslist(args):
    return [ tuple(t) for t in product(*[a.split('|') for a  in args]) ]

class Signatures(defaultdict):
    def __init__(self):
        self.seen = defaultdict(set)
        defaultdict.__init__(self, list)

    def __setitem__(self, key, item):
        if isinstance(key, tuple):
            m, args = key
            for k in argslist(args):self.seen[m].add(k) 
            self[m].append(item)
        else:
            defaultdict.__setitem__(self, key, item)

    def __contains__(self, (m, args)):
        for k in argslist(args):
            if k in self.seen[m]:
                return True

def parse_sigs():
    api = PyQuery(filename='api.xml')
    sigs = Signatures()

    for sig in api('entries entry[type=method] signature'):
        entry = sig.getparent()
        name = entry.get('name')

        for args in arg_combinations(PyQuery(sig)('argument')):
            k, v = parse_sig(name, args)
            v.update({'sig': PyQuery(sig)})

            if k:
                key = (k, tuple(v['types']))

                if key not in sigs:
                    sigs[key] = v

    return dict(sigs)


 'toggle': [{'args': ['handler(eventObject)', 'handler(eventObject)'],
             'name': 'toggle',
             'sig': [<signature>],
             'sig_str': 'toggle(handler(eventObject), handler(eventObject))',
             'types': ['Function', 'Function']},
            {'args': ['handler(eventObject)',
                      'handler(eventObject)',
                      'handler(eventObject)'],
             'name': 'toggle',
             'sig': [<signature>],
             'sig_str': 'toggle(handler(eventObject), handler(eventObject), handler(eventObject))',
             'types': ['Function', 'Function', 'Function']},
            {'args': [],
             'name': 'toggle',
             'sig': [<signature>],
             'sig_str': 'toggle()',
             'types': []},
            {'args': ['duration'],
             'name': 'toggle',
             'sig': [<signature>],
             'sig_str': 'toggle(duration)',
             'types': ['String']},
            {'args': ['callback'],
             'name': 'toggle',
             'sig': [<signature>],
             'sig_str': 'toggle(callback)',
             'types': ['Callback']},
            {'args': ['duration', 'callback'],
             'name': 'toggle',
             'sig': [<signature>],
             'sig_str': 'toggle(duration, callback)',
             'types': ['String', 'Callback']},
            {'args': ['duration', 'easing'],
             'name': 'toggle',
             'sig': [<signature>],
             'sig_str': 'toggle(duration, easing)',
             'types': ['String', 'String']},
            {'args': ['duration', 'easing', 'callback'],
             'name': 'toggle',
             'sig': [<signature>],
             'sig_str': 'toggle(duration, easing, callback)',
             'types': ['String', 'String', 'Callback']},
            {'args': ['showOrHide'],
             'name': 'toggle',
             'sig': [<signature>],
             'sig_str': 'toggle(showOrHide)',
             'types': ['Boolean']}],

I haven't written tests (in javascript) but those sigs look fairly legit

Method (Overload) Madness

2011-01-04T17:03:00.000-08:00

    _toggle: jQuery.fn.toggle,

    toggle: function( fn, fn2, callback ) {
        var bool = typeof fn === "boolean";

        if ( jQuery.isFunction(fn) && jQuery.isFunction(fn2) ) {
            this._toggle.apply( this, arguments );

        } else if ( fn == null || bool ) {
            this.each(function() {
                var state = bool ? fn : jQuery(this).is(":hidden");
                jQuery(this)[ state ? "show" : "hide" ]();
            });

        } else {
            this.animate(genFx("toggle", 3), fn, fn2, callback);
        }

        return this;
    },


....

    animate: function( prop, speed, easing, callback ) {
        var optall = jQuery.speed(speed, easing, callback);

        if ( jQuery.isEmptyObject( prop ) ) {
            return this.each( optall.complete );
        }


....

jQuery.extend({
    speed: function( speed, easing, fn ) {
        var opt = speed && typeof speed === "object" ? jQuery.extend({}, speed) : {
            complete: fn || !fn && easing ||
                jQuery.isFunction( speed ) && speed,
            duration: speed,
            easing: fn && easing || easing && !jQuery.isFunction(easing) && easing
        };

        opt.duration = jQuery.fx.off ? 0 : typeof opt.duration === "number" ? opt.duration :
            opt.duration in jQuery.fx.speeds ? jQuery.fx.speeds[opt.duration] : jQuery.fx.speeds._default;

        // Queueing
        opt.old = opt.complete;
        opt.complete = function() {
            if ( opt.queue !== false ) {
                jQuery(this).dequeue();
            }
            if ( jQuery.isFunction( opt.old ) ) {
                opt.old.call( this );
            }
        };

        return opt;
    },

It's interesting. I thought they would have a more sophisticated and abstract dispatch method. Something declarative and usable by plugins.

I was hoping to be able enumerate the different valid signature types but I have a feeling there isn't going to be some 'easily' exploitable pattern. The args aren't optional in a left to right manner such that for all optional args all args to the left have been optioned. You can call toggle with a callback function as the sole argument. Complicating things somewhat more is the fact that some arguments are described as accepting multiple types. Duration can be a number or string.

fadeTo Shenanigans

Yet the following works perfectly

$(this).fadeTo(function(){ console.log('weird')}, Math.random());

Not that it would be a good idea to rely upon implementation quirks

I'm just interested in automatically computing the different valid combinations of working signatures. Should be doable. Working left to right, taking the first combination of each signature. Cheating for a second and altering my routines to swap out things like "String|Number" to "String" I can see I'm on the right path


 'toggle()': {'args': [], 'name': 'toggle', 'sig': [], 'types': []},

 'toggle(Boolean)': {'args': ['showOrHide'],
                     'name': 'toggle',
                     'sig': [],
                     'types': ['Boolean']},
 'toggle(Callback)': {'args': ['callback'],
                      'name': 'toggle',
                      'sig': [],
                      'types': ['Callback']},
 'toggle(Function, Function)': {'args': ['handler(eventObject)',
                                         'handler(eventObject)'],
                                'name': 'toggle',
                                'sig': [],
                                'types': ['Function', 'Function']},
 'toggle(Function, Function, Function)': {'args': ['handler(eventObject)',
                                                   'handler(eventObject)',
                                                   'handler(eventObject)'],
                                          'name': 'toggle',
                                          'sig': [],
                                          'types': ['Function',
                                                    'Function',
                                                    'Function']},
 'toggle(String)': {'args': ['duration'],
                    'name': 'toggle',
                    'sig': [],
                    'types': ['String']},
 'toggle(String, Callback)': {'args': ['duration', 'callback'],
                              'name': 'toggle',
                              'sig': [],
                              'types': ['String', 'Callback']},
 'toggle(String, String)': {'args': ['duration', 'easing'],
                            'name': 'toggle',
                            'sig': [],
                            'types': ['String', 'String']},
 'toggle(String, String, Callback)': {'args': ['duration',
                                               'easing',
                                               'callback'],
                                      'name': 'toggle',
                                      'sig': [],
                                      'types': ['String',
                                                'String',
                                                'Callback']},

I could make a special class for the the types currently represented by a string sourced directly from the xml. This could compare "String|Number" == "Number" as true. Then tuples could be compared ('String|Number', 'Callback') == ('Number', 'Callback')

5 minutes of fame

2011-01-04T05:10:00.000-08:00

Don't really know how much you can really read into an online test, but seeing I did well I'll not criticize it too much :) Not bad after such a big break.

The pay on oDesk is pretty lousy but it's all relative. If you earned 10 dollars an hour and lived in Cambodia? You can live in a resort on the beach for 500 a month there and it gets a lot cheaper than that ...

Brushing Up

2011-01-04T05:04:00.000-08:00

SRS, jQuery

2011-01-04T02:16:00.000-08:00

Back

My dear readers will be glad to know that I'm back. Where the hell have I been I imagine them asking ( I must have a great imagination, considering noone reads this blog! )

For the last year or so I've been studying music, one of the great unrequited loves in my life. I realised, or shall I say, I came to an even deeper understanding of, how profoundly untalented a musician I am. This is not to say that I don't believe I have potential but I'm positive it will take a Bruce Lee like zeal to unleash and heretofore unseen training methods. I haven't given up. I'm just taking a timeout to develop more programming skills to apply to the problem.

Throughout the year struggling away at ear training, using the available software I found myself yearning for something better. Being one of those arrogant "I could do this better" programmer types cursed also with a low pain threshold ("I have to use a mouse?? kill me!") ideas started to form in my mind. I didn't have the time or inclination to work on any of them however as I was busy studying and pretty much burned out on anything IT related.

But that's never here nor there. The central topic of this blogpost is 'scheduled repetition software'. Why lead in with your mediocre musicians lament?

Well, if you ever *completely stopped* something for any length of time you'll know the danger of gaining rust and losing insights. In the Skydiving world they call this state 'uncurrent' and you are considered a danger to yourself and others if you go for longer than a few months without jumping. We've all heard the "Use it or lose it" maxim.

Having finished up the year of study, I realised I needed some work and decided I should brush up on some of my old programming skills. Ouch! I've been somewhat blessed with a pretty good long term memory and retention. However, there's so much I just plain forgot.

Little details that before were readily available were now lost leaving me no recourse but to use references. "Where do I import that module from again?" "How do you get hg to automatically update server side?" "What was the keybinding for that command?" "The trigger for that snippet?" "What was the difference between left outer joins and ... "

And so on.

At one point during the year, I actually memorized 72 ascending musical intervals over four or five days. You think I could recall them a month later? No. Cramming doesn't work. You need to review. I can recall the intervals in the key of C and some in G/D/E but most I've forgotten.

Now I was considering studying this year also. It occured to me that it would be hard to maintain a decent skill/knowledge level to be able to work efficiently enough that I wouldn't be better off just digging ditches.

I knew it would be futile to try and cram in a heap of study over the holidays.

  
  TODO:
  Brushup on WebDev

       jQuery in action.
       CSS, The Missing Manual.
       Firebug, Beginners Guide
       SQLAlchemy tutorial
       Genshi templating

I've done cramming in the past with things like Django (for an ill fated project ...), and you invariably forget it unless you use it day to day until it's burned into your long term memory.

How to bypass the need for a long period of 'day to day'? How to get something burned into your memory 'unnaturally'?

One thing you learn as a musician, is that you really need to apply 'deliberate practice' to get better. I'm not sure it's something many programmers do. At least I know I never did. In fact even when 'current' there were somethings I routinely used a reference for, even as little as a week between, just because my mind was habitually forgetful.

I left school at 12 years of age, obviously never attending university. Needless to say my study skills aren't very sophisticated. I actually find this thought rather encouraging as I know I'm miles from having plateued. There's lots of "low hanging fruit" ready to be picked.

So I had 6 weeks until school started again (at this point I planned on returning to study). I realised I should probably read into some study tips.

I downloaded a bootleg copy of the Memletics manual, which goes into quite some depth about the most efficent methods for learning.

Nutrition.
Exercise.
Attitude.
SRS.

I've still yet to apply most of what I read in there. If you think turning short term memory into long term memory is hard try developing new habits!

The biggest change in my learning habits is the use of SRS software. In fact I even got in some daily revisions on XMAS day.

Enter Anki

Anki is an implementation ( PyQt ) of software based flashcards. What makes this different to normal flashcards is that it intelligently schedules your cards for repetition at a time when it thinks you'll forget. I guess by now you have gleaned that SRS is an acronym for Scheduled Repetition Software.

The cards are html with all that entails: you can embed sounds/images and hyperlink. It has some other cool features like 'download shared deck'.

In fact, I found that someone had created a jQuery deck with roughly 180 cards.

Now, as I've learned there's something of an art to creating flashcards. You really need to reduce the amount of information to one or two discrete chunks.

This of course violates your prejudices against redundancy. IIRC, and I'm pretty sure I do (at least I remember this!), there was a meme floating about by the name of DRY, "Don't repeat yourself". With flashcards, scheduled REPETITION ones at that, the whole point is for repetition so that's OK.

As Anki is implemented in Python/PyQt and is extensible via plugins, one of the first things I did was to create a PYRO based bridge between it and my python extensible editor.

I'd copy/paste text from pdfs websites into a text buffer, manipulating the information into Q/A form then push 10 - 20 cards at a time using the multiple selection capabilites of my editor.

This information reorganizing process itself is invaluable as it really forces you to go beyond habitual skimming.

This requires some time and discipline and feels like a lot of work so I'd imagine it's not something everyone will have the stomach for. However, 'slow is fast'. What's the alternative? If you spend a few hours skimming and within a few weeks can recall nothing of value you've wasted your time. You have to think long term.

If it's something you think you'll eventually want/need to know then it's a good idea to spend the time upfront. How many times have you wasted countless hours fucking about blind, cause of artificial deadline pressures? I know people who have been developing for ages who always 'too busy' to RTFM.

Software development is complex. How can what is basically glorified rote memorisation really help?

How can you make critical implementation decisions if you don't know the frameworks you are using? How many times have you reinvented the wheel, because of lack of knowing your options? How much more creative can you be if you really know your 'material' and can think clearly about possibilities? It's what makes an expert and expert.

I applied Anki to brushing up on CSS, jQuery, Firebug and am learning the PyQt framework. At the rate I crammed some of it in, I doubt I'd have retained much without Anki.

jQuery Examples

One of my projects I've been tinkering on lately is a flash card export of the jQuery API raw xml dump. I wish to create flash cards going from {short_description : method_name} and its inverse {method_name, short_description}

Not only that but I want each html card to contain links which open up the example code in my editor. Luckily my editor can run commands argumented via the command line. I created a custom sblm:// protocol which allows commands to be sent in the query string.

I created a routine to determine all valid combinations of arguments so I can memorise the signatures of routines and create snippets. ( I'll have to make it a little smarter to filter the conflicting variations somehow. There's only so many ways a function can interpret a single boolean for instance )

    def arg_combinations(args):
        args_n = (int(bool(e.get('optional', False))) for e in args)
    
        for n in xrange(2 ** sum(args_n)):
            opt_args = []
            n_optional = -1
    
            for arg in args:
                optional = arg.get('optional')
                if optional: n_optional += 1
    
                if not optional or optional and n & (2 ** n_optional):
                    opt_args.append(arg)
    
            yield opt_args

 'toggle()': {'args': [], 'name': 'toggle', 'sig': [<signature>], 'types': []},

 'toggle(callback)': {'args': ['callback'],
                      'name': 'toggle',
                      'sig': [<signature>],
                      'types': ['Callback']},
 'toggle(duration)': {'args': ['duration'],
                      'name': 'toggle',
                      'sig': [<signature>],
                      'types': ['String|Number']},
 'toggle(duration, callback)': {'args': ['duration', 'callback'],
                                'name': 'toggle',
                                'sig': [<signature>],
                                'types': ['String|Number', 'Callback']},
 'toggle(duration, easing)': {'args': ['duration', 'easing'],
                              'name': 'toggle',
                              'sig': [<signature>],
                              'types': ['String|Number', 'String']},
 'toggle(duration, easing, callback)': {'args': ['duration',
                                                 'easing',
                                                 'callback'],
                                        'name': 'toggle',
                                        'sig': [<signature>],
                                        'types': ['String|Number',
                                                  'String',
                                                  'Callback']},
 'toggle(easing)': {'args': ['easing'],
                    'name': 'toggle',
                    'sig': [<signature>],
                    'types': ['String']},
 'toggle(easing, callback)': {'args': ['easing', 'callback'],
                              'name': 'toggle',
                              'sig': [<signature>],
                              'types': ['String', 'Callback']},

I wonder if you really can pass a single string as an arg and have the toggle function determine whether you meant duration or easing? I'll have to try it our or better yet, look at the jQuery soruce

For purposes of automatically creating snippets I guess you really only want to look at the different signature types. jQuery uses anonymous functions a lot and literal string based arguments so creating snippets with placeholder args already quoted and function(){} inline seems like a decent idea

Web Snippets

There's a reason iterative development is usurping upfront design methodologies. We are fallible and mostly lacking detailed imagination. Our most glaring and atypically ungrudging admission: the fact we are so obsessed with testing.

I'm a great believer in efficient text editing. This shortens your feedback loop. If you can input text twice as fast you get through, I dunno, probably not twice as many, but a lot more 'fuckups'. Iterations!

The thing about snippets I've found is that unless I can recall instantly the abbreviation, I'll tend to just type in the whole words.

Now this really ties back to what I was saying before about 'deliberate practice'

In theory, if you practiced enough you could use completely arbitrary 2 letter combinations to compress an API worth of words down. However, memorising, even with SRS, is easier with some mnemonics so...

I've actually written some routines which will take a list of words and attempt to create ideal abbreviations and ensure unique.

One of the routines uses the Carnegie Mellon University Pronouncing Dictionary, cmudict for short, found in the nltk toolkit.

    def rank_letters(wd):
        if not wd in CMU: return []
    
        cmu = ''.join(['_' if ch[-1].isdigit() else ch for ch in CMU[wd][0]])
        cmus = defaultdict(int)
    
        mapping =  '^(\\w)_|_(\\w)_|(\\w)_|_(\\w)$|_(\\w)'
    
        for i, w in enumerate(mapping.split('|')):
            for m in re.finditer(w, cmu):
                ch = m.group(1).lower()
                if ch in wd: cmus[ch] = max(i+1, cmus[ch])
    
        return sorted(cmus.keys(), key=lambda k: (cmus[k]))

Intuitively I sensed that words surrounded by vowel sounds are the 'strongest' letters in a word, followed by those adjacent to only one. It seems to work reasonably well for such a crude method.


eg
    document - DocuMeNt - dm
    border   - BorDer   - bd

I also used the words corpus to create routines to decompose composite words.


    mouseup    -  mouse, up     -  mu
    mousedown  -  mouse, down   -  md

I'm going to create desc/entity flashcards and snippets also for html elements/attributes and css property/values.

With my editor at least, the bindings are smart enough to know when the cursor is inside a tag. eg. <|div> Why the hell type in annoying long character sequences when the set of identifiers compresses down to one, two, maybe three (ouch) character ids?

Over a course of weeks, you could, with SRS, systematically memorise the entire html set of elements and their attributes along with related snippet abbreviations. Shld I sy abrv instd?

It would be analogous to touchtyping. A lifetime of benefit for a short period of practice.

On second thought ... A lifetime of entering html/css? There's a horrid thought...

Statistics

Ideally I'd like to give the most frequently used entities the most suitable and short abbreviations, one character ones at times.

Google apparently did a massive study looking at web authoring statistics. They have the data as to which elements should receive the honorary 1 keyers.

Namespaces

You mean there's other javascript snippet candidates other than jQuery? Poses some problems. Where there's a problem, there's a solution.

Consolation: It's not you, it's the other guy.

2009-07-28T22:25:00.000-07:00

I have been doing some freelancing work but it hasn't really been buttering the bread on a consistent basis. I avoid PHP which doesn't really leave much else job wise. Normally I value my time and flexible working hours far more than dollars so a full time job isn't something I'd really look for. Once you have been poor long enough however you start resorting to desperate measures and thinking of all sorts of crazy ideas. Boredom sets in and you even do things like offer your services practically free. Fail with consolation:

it is also clear to me that you possess attributes that would make an excellent employee - talent, enthusiasm and dedication. I would have no issues with offering you work if I were able to get XXXX to that stage... I am more than happy for you to put me down as a reference for any job opportunities, as I would highly recommend you to any employer.

Now whether the guy was just politely fobbing me off or he actually meant that I'm not entirely sure half the time. I've never particularly thought of myself as `talented` when it comes to code. `Persistent` like a retarded dog who doesn't know when to stop, sure.

When you can't practically give yourself away and you again decide you really want stable cash what do you then? 'I know! I'll look for a *well paying* job. What a shrewd and cunning plan!'

As I'm starting to get reasonably fluent with python, not needing references every 10 seconds, I naturally looked for jobs using that. There were however none available that didn't require `3 years experience` and `knowledge of finance` etc. The `python` keyword did however turn up another interesting advertisement.

We're a Richmond-based software development studio specialising in web technologies such as Flex and Ruby on Rails. We are looking for a passionate and energetic developer to join our team of three.

We're more interested in what you can do than what you say you can do. So prove yourself with your attitude and with your code.

We're looking for someone who's spent their evenings and weekends working on their own projects, and has something to show for it. We have a particular love of open source collaboration, and you should too. Tell us about the projects you watch, and the projects you contribute to: do you have accounts with Github, SourceForge, or RubyForge?

We want to be floored by your enthusiasm to work with new products, programs, and languages. Dynamic or functional languages like Python, Ruby, Actionscript, Erlang, and Haskell are looked upon favorably. We're not interested in your academic transcript, your 2-day ScrumMaster black-belt, or your Microsoft certificates, but we are interested in real talent, sharp skills, and unmatched motivation.

You must send through a cover letter describing your pet projects or open source projects, example source code from the project, and your resume to recruitment@silverpond.com.au. We won't consider any applications which don't have all three.

Normally I wouldn't have even bothered applying for something like that but I thought, if that guy will give me a reference then I may as well try it. I didn't really have much time to create an application or research the position all that much as I was away on a trip helping my father with a building project. Being an unfit person who would normally spend way too much time on a computer, after a days work I was quite tired in my poor fitness. I mostly reused an old application for another Job at a game development firm.

I was somewhat suprised when I got a positive sounding response from the recruiter/director of the company "I'm free to chat late this week or early next week, so I'm looking forward to hearing from you"

We eventually had a chat a few days later, which I wasn't exactly well prepared for. There was some initial awkardness with some nerves causing to me blurt out "good thanks and yourself" completely out of phase. It then smoothened out somewhat and I was happy to find the team was a group of 20 somethings and that it wasn't an issue I didn't have any experience with the technologies they used. I didn't really have many decent questions prepared ( it was actually my first job interview! ) but regardless at the end I was left with the impression it went well enough. The recruiter wanted me to come in a few days later for some facetime with the rest of the team.

As it turns out I never got that opportunity. Someone else had done their homework and really impressed them so well with their phone interview they got the job solely on that. I found this out as after as I asked the recruiter for any particular reasons the other candidate was chosen (over me). I was very appreciative with the email he replied with:

1) The candidate we chose asked questions about the projects we're doing and suggested improvements to them

2) The candidate also outlined what he could contribute to the team, and offered a timeframe to do them in.

Of the 80 of so applicants it was between the candidate, one other person and yourself. Your energy and enthusiasm is highly commendable. I'll touch base with you later in the year, say November, and see where you're at.

Overall, I have mixed feelings. I'm somewhat disappointed that I didn't have greater time to research the company more and also that I didn't make better use of the time I did have. The biggest disappointment is the fact that I missed out on a job where they didn't care that I lacked a degree or even any real experience with their tools of choice. "I don't really know ruby.." I had said. "It won't be an issue" was the reply. Similar conversations re: git / rails / flex. However it's somewhat encouraging to have made the short list from 80 candidates. That I feel pretty good about.

Will I continue investigating ruby / rails? My only concern is that it seems somewhat wasteful to walk away from my python experience ( It's the libraries stupid ) for a language so similiar. My `next language` to learn has always been planned to be C then possibly assembly.

I wonder if I would learn much from ruby about programming in general coming from python. Perhaps the similarities would be actually a boon to recognizing core concepts and distilling an understanding, compared to something radically different where you have no reference? I'm not really sure and I doubt I could really know for myself unless I tried. However, at the end of the day; there *does* seem to be a lot more work available for ruby/rails as opposed to python.

I can't shake the nagging suspicion I have just been "It's not you it's XXXX"d twice.

I posted on facebook, "didn't get the job and wonders why he marginalizes himself learning obscure tool-sets". A friend commented:

And don't focus on tool-sets - focus on your general problem solving abilities, There are a million different ways to solve any programming problem these days. Key thing is to be adaptable.

I agree with the notion that general problem solving ability is more important than any particular toolset. However, if you had said ability *and* a proficiency with a particular toolset it would make sense to use them where possible. Using the `right tool for the job` is part of solving problems.

Python isn't exactly an `obscure` toolset, however if you are unwilling to work with PHP you definitely are marginalized. I have a very low pain threshold in general. PHP is painful. I don't like pain. Nuff said.

Of interest to me was that the recruiter during his `wish you all the best` phonecall mentioned that the ruby community met once a month (in my home city) Perhaps learning some ruby/rails and getting involved would at the least yield some freelancing contracts. Never really been to any geek meetups. Surely beer is involved?

Ruby Tuesday

2009-07-27T23:18:00.000-07:00

IPython is a huge part of my workflow. How to replace it?

(1:24:57 PM) akalias: Does ruby have an equivalent to python's IPython ? ie a hugely enhanced shell with better completion / syntax coloring etc?
(1:25:32 PM) dominikh: irb with auto_indent and the "wirble" gem

I have sourced a few ruby/git ebooks

Windows Ruby Solutions

2009-07-27T20:39:00.000-07:00

(1:26:08 PM) akalias: what is the best option for ruby on windows? a vm with linux ;) ?
(1:26:25 PM) dominikh: in my opinion actually yes...
(1:26:39 PM) dominikh: don't know if you can even get a decent terminal in window

The solution is simple: remove windows from the ruby mix

I have VMWare installed and preconfigured 256MB `headless` Ubuntu webserver appliances are readily available. If I set up one of these with SAMBA file-sharing I should be able to use my existing toolset quite easily and use SSH to log in.

Now if only the internet wasn't so damn slow where I live...

UPDATE:

Net is super slow - however I discovered that I had an old Gutsy Gibbon image laying about on one of my portables. It already had colemak keymap installed. The problem was Gutsy Gibbon has reached the EOL. apt-get was no longer working

(5:00:13 PM) ubottu: wants you to know: For upgrading, see the instructions at https://help.ubuntu.com/community/UpgradeNotes - see also http://www.ubuntu.com/getubuntu/upgrading

No problem.

UPDATE:

Compiling Ruby 1.9.1 from source

Windows Ruby Options

2009-07-27T19:22:00.000-07:00

I'm looking at learning ruby/rails/git for a full-time job I'm a candidate for. Windows seems to be very much a second class citizen in all of these worlds. Git at least has MSYS git readily available and well maintained. Ruby is another story.

The 1.9 version of ruby, I'm assuming is used by the company, was released in late 07 and now, midway through 09, there is no `one click installer` available.

You can download a 1.9 binary zip at patch level 0 ( the latest is in the high two hundreds ) but it is missing crucial dlls such as zlib and readline.

What's a windoze fool to do? I'm familiar with linux, mostly ubuntu/debian but a lot more productive in a patched up windows. I use replacement windows managers, independent virtual desktop managers for each monitor, a replacement for explorer etc

I have tried setting up a similar environment using linux before but my crappy video card wouldn't allow virtual desktop sets per monitor. Changing environments *and* languages at the same time wouldn't be that smart.

Probably the best would be to try ( in order ):

Find native windows versions of 1.9
Set up a VM running ubuntu and file-sharing/ssh
Run linux natively
Run a mac

I have played with ruby under cygwin in the past and it was retarded slow so that is not really an option.

Delving Into Ruby

2009-07-27T18:29:00.000-07:00

A lot of python users seem to hold a fair amount of contempt for ruby. They don't like the use of `perlish` punctuation and operators like `@`, `=~` and `:` and its `more than one way` expressiveness.

I have always looked at ruby code and thought the syntax seemed quite readable. Having explicit repeated sections like `self.` as instance variable in python seems just noisy and I prefer the more compact @ruby way.

Rails seems to be architected with a controller class `instance` per request, the view context sourced from the controllers instance variables.

def method
    @a = 'hello'
    @b = @a * 4
    @c = @b * 3
end

Compare this with the typical python web paradigm. Instance variables (self.xxx) are unused as controller instances are shared between requests. Locals are built up first and then redeclared, returned as a context dict.

def method(self):
    a = 'hello'
    b = a * 4
    c = b * 3

    return dict( a=a, b=b, c=c )

The example is very basic but note in python you couldn't even validly do the following

def method(self):
    return dict (
        a = 'hello',
        b = a * 4,
        c = b * 3,
    )

`a` hasn't been declared until the dict structure has been parsed therefore you can't reference it to build `b`. At best you can do

def method(self):
    a = 'hello'

    return dict (
        a = a,
        b = a * 4,
        c = b * 3,
    )

I have always hated that pattern (as do some other pythonistas) and would generally return locals()

def method(self):
    a = 'hello'
    b = a * 4
    c = b * 3

    return locals()

But what if you want to convert your context to JSON? At the very least you have the `self` controller instance in your context. This doesn't convert very well to JSON literals. You then have to filter keys from your context dict. It's supposed to be the Python Way to be explicit ( ie declaring your context vars rather than using locals() or horrible _getframe hacks to try and remove some repetition )

Beautiful is better than ugly.
Explicit is better than implicit.

To me, being explicit in Python (in this case) breaks the first commandment; it is fugly. Contrastingly, the compactness of the @ruby way, a trifling detail, allows a quite sweet pattern. Just prefix any variables you want in the context with a @. I appreciate it not just from a purely visual sense of aesthetics; practically it would be easier to work with.

How on earth is using @ here less readable or uglier on a whole? Could it be better? maybe .

Anyway, `It's all subjective`, `Beauty is in the eye of the beholder` and `Ugliness is to the bone` and other cliches.

NodeSelect (or Hi SilverPond!)

2009-07-19T00:43:00.000-07:00

I haven't blogged in a month or so as generally I prefer actually coding to blogging about the coding I am doing. So many ideas! So little time! I have noticed quick screen-casts illustrate concepts much more effectively and efficiently. Therefore when communicating about projects with fellow programmers I'll generally just whip up a quick cast.

So what's been up? Amongst other fun projects, lately I have been working on creating some xml/xhtml helpers for my editor. It's still in prototype stage but the basic concept is proving to be quite promising. As I'm away on a trip I'll just list the commands and link to a few screencasts recorded earlier for the moment.

This quick blog post expressly for the purpose of showing a potential employer some of my `pet projects`.

Commands

pathSelect CSS
pathSelect XPath
collapseNode
prettyPrintNode
commentNode
tidySelection
selectInsideTag
moveToNodeEnd
selectElementName

ScreenCasts

Sprox

2009-05-29T06:25:00.000-07:00

An interesting CRUD library based on sqlalchemy / formencode / toscawidgets. It actually generates the html. Looks quite handy if you wanted fully generated forms.

Sprox

I'm not sure if it can do nested model forms or if it can that the functionality is decoupled from the html generation and available for use. It seems like the author has put a lot of thought into it. I will definitely investigate it and ToscaWidgets further. I haven't really had much experience with auto-generated forms. I get the feeling that they are hard to customize but it's likely worth auto-generating when possible.

There doesn't seem to be m?any projects focused on (semi or fully) automatically creating list/pagination filter forms. This is quite commonly required in admin sections of a site.

I can't even remember if Django does.

sqaencode 0.1

2009-05-29T01:55:00.000-07:00

I have been messing around a bit and now pretty much have a working lib for encoding nested models into forms and back and automatically generating formencode Schema from sqlalchemy models.

E:\python_repos\sqaencode>nosetests --with-coverage --cover-package=sqaencode
................
Name                   Stmts   Exec  Cover   Missing
----------------------------------------------------
sqaencode                  6      6   100%
sqaencode.constants       10     10   100%
sqaencode.decode          47     45    95%   126, 161
sqaencode.encode          28     26    92%   90-91
sqaencode.util            12     12   100%
sqaencode.validators      46     42    91%   30, 64, 66, 129
----------------------------------------------------
TOTAL                    149    141    94%
----------------------------------------------------------------------
Ran 16 tests in 0.962s

OK

I created a ModelSchema class, subclassing formencode.Schema and giving it an inline __metaclass__ inheriting from formencode.declarative.DeclarativeMeta. I over-rode the __repr__ to output an (almost) eval()able representation.

class ModelSchema(Schema):
    class __metaclass__(DeclarativeMeta):
        def __repr__(cls):
            model = cls.__model__.__name__
            base = [ "class %(model)sSchema(model_schema(%(model)s)):" % dict (
                              model = model)  ]

            for arg in SCHEMA_ARGS:
                base.append('    %-20s = %r' % (arg, getattr(cls, arg)))

            base.append('')
            for key, validator in sorted(cls.fields.items()):
                if key.startswith('_'): continue

                args = non_default_validator_args(validator)
                base.append('    %-20s = %s(%s)' % (
                    key, type(validator).__name__, args) )

            return '\n'.join(base)

Using the model_schema factory to create a ModelSchema and then getting a repr:

In [1]: from sqaencode import model_schema

In [2]: model_schema(Product)
Out[2]:
class ProductSchema(model_schema(Product)):
    ignore_key_missing   = True
    allow_extra_fields   = True
    pre_validators       = []

    active               = Bool()
    amount               = UnicodeString()
    colour               = UnicodeString()
    description          = UnicodeString()
    featured             = Bool()
    id                   = Int()
    image                = UnicodeString()
    image_thumb          = UnicodeString()
    image_zoom           = UnicodeString()
    keywords             = UnicodeString()
    material             = UnicodeString()
    name                 = UnicodeString()
    ordernum             = Int()
    sku                  = UnicodeString()
    views                = Int()

The __metaclass__ __repr__ hack serves a dual purpose. a) for debugging and b) as templating. By printing the repr you can use that as a template for customizing a model schema. You actually inherit from a dynamically generated class ( a function call taking a model and optional arguments). I wasn't even too sure you could do that in python. It's nice I don't have to create a new meta class mechanism and can stick with existing formencode semantics.

Note the `ignore_key_missing` flag that is by default set to True. I think when using this I will just define which fields to validate purely by virtue of what is included in the html. eg If there is no `sku` field in the form then it will not be validated.

What if I *didn't* want to globally ignore missing keys and wanted to manually declare which to ignore? Formencode has an in-built sub-classing mechanism whereby if you declare `some_key = None` then some_key will not be validated at all.

To declare a Product model_schema with plural Colors inline:

class ProductSchema(model_schema(Product, nested=True)):
    colors = model_schema(Color, plural=True)

With Python 2.6 you can do inline customisable declarations of relations:

class ProductSchema(model_schema(Product, nested=True)):
    @sqaencode.plural
    class colors(model_schema(Color)):
        some_field = NonDefault()

I'm thinking about creating a mechanism whereby I subclass sqlalchemy.types.* for metadata purposes to further drive automatic schema generation. A cool thing about Django's tight integration is the high level data types. Url, Email etc You declare higher level properties to what are essentially stored as VARCHAR types in the database. It's not *just* a string.

SqlAlchemy, while really great, (rightly) doesn't try and abstract beyond basic SQL. There is nothing stopping an end user however doing something like this:

1  class Url(Unicode):   pass
2  class Email(Unicode): pass
3  
4  higher_level_table = Table ( 'higher_level', metadata,
5      Column('url', Url(32)),
6      Column('email', Email(32)),
7      Column('id', Integer(), primary_key=True, autoincrement=True, nullable=False),
8  )
9

If Url and Email were imported column types from sqaencode.types then you could add them to the sqalchemy => formencode type mapping and they would be picked up by model_schema()

What's on the todo?

Options for automatically generating child Schemas.
Setting the length on String/UnicodeString validators automatically to the max length of the corresponding column.
Automatically create unique column validators

Reading

2009-05-27T22:20:00.000-07:00

A few days ago I scribbled an entry into my TODO.txt:

Learn about Python packaging, namespaces etc, best practices. setup.py etc etc

I started reading a lot and watching whatever screen casts I could. One useful book was Expert Python Programming. It is light on actual python and heavy on the soft skills in using open source python tools to refine your work flow. That is exactly what I was looking for. It is only about 300 pages so I skimmed through most of it in a day.

The Agile Development Tools in Python series by Christopher Perkins, while pretty lighton gave some nice quick overviews of some very useful tools.

Between these two sources I got a good overview of:

setuptools
distutils

virtualenv
paster

nosetest
coverage

docutils
reST
sphinx

virtualenv

virtualenv will set up an isolated version of python, where applications can remain blissfuly ignorant of the rapid change in the outside world. If you have an application that needs a particular version of a lib and another requiring a different one then this is the tool for the job.

It will set up a folder structure with (std) Lib and site-packages folders and a scripts directory. One of the scripts is of course the virtual python interpreter. It also has a tremendously useful `activate` script. This will put the Scripts (bin on *nix) folder on PATH and the lib folders on PYTHONPATH. It also modifies your console prompt to display the name of the current virtualenv.

(myenv) C:\myenv

I haven't used it much as of yet but I can imagine that it could get pretty confusing once you had a tonne of them so that is a nice and thoughtful addition.

The net result being that if you `activate` an environment and then run `python some_python.py` from an arbitrary directory it will run it using the virtualenv. Likewise for any of the scripts in your (*nix: bin, win: Scripts) folder. There is no need for explicitly referencing them. When you are done you just `deactivate` the environment with corresponding script.

setuptools is installed by default into each new virtualenv and easy_install is included in the scripts folder. You use easy_install to, funnily enough, easily install the packages you require.

virtualenv seems of great utility, however I imagine it could get pretty wasteful in terms of network usage. You don't really want to be downloading 10 packages EVERY time you start some new project. It would be handy if all of your local virtualenv environments used some sort of global local cache when using easy_install.

If you needed for example cherrypy and you already had the latest version on your hide drive some where it would just pull the egg from there. Otherwise it would first pull the package into the cache from pypi.

It would be good if this all happened transparently and each virtualenv's easy_install respected some global distutils.cfg setting for cache location. Where do eggs lay about usually? In a Nest. I'll probably happily find that this has been sorted out.

Even if you had a cache, saving yourself some bandwidth, if you just copied an egg into each environment that would still be disk wastage. This seems like a good use for symbolic linking. It would be silly to reinvent that in python. Although, I'm not too sure how well Windows supports something like that. *nix is definitely superior in that respect.

paver

Another interesting util is one called paver that sort of ties together virtualenv and distutils/setuptools to allow a more `pythonic` zc.buildout. Buildout uses declarative ini files for everything and is apparently harder to hack if you want to do anything out of the ordinary. (I wouldn't know for sure if these are valid criticisms )

From what I have read of buildout vs paver/virtualenv I think I will invest in the latter. It just seems to appeal more to my personal sense of aesthetics.

paster

Paster is a hodgepodge macgyver swiss knife created by Ian Bicking. It can launch WSGI applications, launch missiles, sink ships and god knows what else. This utility, `eclectic` in the words of Ian, is basically the kitchen sink.

Of interest to me is the project template creation commands. It will create a folder with setup.py, README.txt etc and run you through a command line based wizard to set it all up, asking version number, author name and the like. These project templates can give a bit of an insight into how other people structure their projects.

Where do the tests go? Inside the distribution proper? Housed inside the same folder as the setup.py? I'm somewhat leaning towards having the tests included in the public package. ie package.tests.fixtures I think it makes it easy then to import fixtures in doctested examples documentation.

distutils / setuptools

With regard to structuring projects, if there is one thing I can hope to take away from all this, then it is the `setup.py develop` command.

This distutils (or is it setuptools?) command will put your under development package on sys.path so you don't have to run `setup.py install` every time you make changes. Quite handy. Of course you could have a virtualenv where you have a package in `develop` mode and another where you have ran a full `setup.py install` at a certain version.

nosetests

Nosetests has a really great coverage plugin. It will run your tests, and you can specify which package/module you want it to dump a list of line ranges that haven't been `covered` by your test code. This makes it super easy to tell which code paths still needs testing( or culling! ). I'm assuming this is only really of use for pure python code. If so, that is another tick next to dynamic `interpreted` languages.

sphinx

I also experimented with docutils/ reST / Sphinx for documentation. Sphinx is the spiffy new reST based documentation system developed originally for Python proper. It has a built-in indexer and javascript search client and seemingly many other great features. For this reason it seems a LOT of python projects are using it now. I imagine the fact that it's quite easy on the eye out of the box doesn't hurt take-up much either.

A new project

I started on creating a setup.py enabled, nose/doc tested, sphinx documented project. It *really* slows down the whole process. I suppose it's a lot to try and learn all at once.

I'm still not really sold on test first prototyping. TDD of something you have a little bit of experience with, sure. TDD/documenting something when you are in new terrain and you KNOW that you are going to mess up and have to redux anyway, just seems wasteful.

I probably just don't `get it` yet. `Slow is fast` has generally held true in a lot of other areas of my experience.

What else to read up on? I need to learn more about the standard library logging package.

Souvenir

2009-05-22T22:39:00.001-07:00

prison color blue
it's a uniform of choice
count yourself lucky
that you don't write the software

Some lines from the great Neil Finn's song `Souvenir`. Count yourself lucky indeed.

FormEncode + SqlAlchemy = SQAEncode

2009-05-22T20:13:00.000-07:00

The way I do admin model CRUD at the moment is have a models.py file containing all my sqlalchemy tables/classes and their respective formencode.Schema/s in a forms.py file. The problem is that I end up having to repeat all the fields and to me that just stinks.

It's not a big deal as far as typing them out goes as I will typically declare a table / model in sqlalchemy and then use a homebrewed scaffolding function to generate derived code for the Schema and html for the actual form. When actually editing a model form I use genshi's HTMLFormFiller filter to set the blank forms values.

This approach really is keeping in the spirit of the formencode library. The author's philosophy is that after a while configuring an `autogenerated` form from a model by declaring options in code is just as much work as declaring in html.

Configuration includes black/white listing certain fields, whether they are allowed to be empty (null/None) and also logical/presentational ordering and grouping of individual fields.

formalchemy on the other hand will take a model and automatically generate a form with the values already set. It can not though do nested models. ie A parent model with an arbitrary amount of inline children to CRUD in the same form. The author considers this `almost always bad design` It only allows what I term `relating` the parent/root model to existing models.

While it *may* sometimes save you from doing any `configuration` whatsoever in the case where the form validation schema reflects perfectly the model, in my (admittedly meagre) experience this would be fairly rare.

The authors of formalchemy recommend the use of CSS to customize the appearance of the auto generated forms. For more extensive customization you can override each fields renderer (<input type = 'text'> vs <textarea> etc) or the global form generation function.

Having not really used formalchemy that much I can't say with certainty just how unweildy it gets trying to customize a forms options / looks. I'd guess that you would end up having to do just as much `configuration`, ie repeating of fields / `work`

Looking at a lot of my formencode Schemas in relation to their underlying model, it seems they are declared in a `white list` manner. The schemas (and html) simply don't declare fields that aren't `public`. The underlying type map is reasonably consistent. A sqlalchemy Bool becomes a formencode Bool, a Unicode maps to a UnicodeString etc. It's typically only the options that are changed.

I have an emybronic idea to auto generate formencode validators from an sqlalchemy model, mapping validators to column/relation types. I imagine `configuring` this would look something like the following.

1  class ProductSchema(ModelSchema): 
2      _model = Product
3      
4      long_description = options(not_empty = True)
5      date_created = None
6      name = UniqueName()

The ModelSchema would use some type of __metaclass__. Any `private` field declared as None would be blacklisted from the underlying derived schema. Above `options` would just be an alias for dict. A dict would be used to configure the auto mapped validator. `not_empty = True` would override the not_empty argument mapped from the respective columns nullable property. UniqueName above would completely override from the UnicodeString. (In fact any column with unique = True could probably have an auto generated `unique field` validator attached)

I'm also currently working on some sqlalchemy formencode integration functions that will take basic one table models and their relations and encode them into nested dictionaries / primary key lists. Also, the other way round, taking a nested dict (as from a formencode.NestedVariables pre validated Schema) and creating/updating/deleting an object tree/graph.

SqlAlchemy mappers makes it quite easy to introspect classes for relations so the object_graph func signature just looks like:

1  object_graph(nd_dict, root_model)

This all works fine in basic unit test land AND for one model forms. The problem I'm having is ONETOMANY child objects in a form. Imagine an Invoice form with an inline InvoiceItems table with each row representing an individual item. On a new invoice there would be 3 `blank` rows. How would you know which ones to ignore? Which ones to delete?

I want to set a `_keep` flag for each item. The _keep flag will be reflected as a checkbox in the form and if unchecked will mean that the child item should be ignored in the validation process.

I could leave the child tables ( invoice items in the concrete sense) empty and use javascript to add a [+] button to add new children. That to me seems like a `bent over` concession.

The `object_graph` function which CRUDS model[s] from a nestable dict is currently decoupled from the validation process and knows nothing of form errors. This might make it hard. Before that I was successfully using `child_crud` hooks in my CRUD controllers for this purpose.

I'll have to sit down and work out which cases I'll need to accommodate. I want to be able to update/delete existing models, create new models and ignore cases where the child CRUD form partials have not been edited.

How to get the formencode.ForEach validator to ignore items without messing up the errors index? These problems were all solved using a child_crud hook in the controllers but trying to abstract and decouple things is making it a lot harder.

It will be a nut worth cracking anyway.

PyParsing + SqlAlchemy = Basic Search Engine

2009-05-11T06:55:00.001-07:00

Today I learned about writing recursive descent parsers using the PyParsing library. I managed to cobble together an sqlalchemy expression builder for a basic search engine.

 1  #################################### IMPORTS ###################################
 2  
 3  # PyParsing
 4  from pyparsing import ( CaselessLiteral, Literal, Word, alphas, quotedString,
 5                          removeQuotes, operatorPrecedence, ParseException,
 6                          stringEnd, opAssoc ) 
 7  
 8  # SqlAlchemy
 9  from sqlalchemy import and_, not_, or_
10  
11  ################################## LIKE ESCAPE #################################
12  
13  LIKE_ESCAPE = r'\\'
14  
15  def like_escape(s):
16      return '%' + ( s.replace('\\', '\\\\')
17                      .replace('%', '\\%')
18                      .replace('_', '\\_') ) + '%'
19  
20  ############################### REUSABLE ACTIONS ###############################
21  
22  class UnaryOperation(object):
23      def __init__(self, t):
24          self.op, self.a = t[0]
25  
26      def __repr__(self):
27          return "%s:(%s)" % (self.name, str(self.a))    
28  
29      def express(self):
30          return self.operator[0](self.a.express())
31  
32  class BinaryOperation(object):
33      def __init__(self, t):
34          self.op = t[0][1]
35          self.operands = t[0][0::2]
36  
37      def __repr__(self):
38          return "%s:(%s)" % ( self.name,  
39                               ",".join(str(oper) for oper in self.operands) )    
40  
41      def express(self):
42          return self.operator[0](*( oper.express() for oper in self.operands ))    
43  
44  class SearchAnd(BinaryOperation):
45      name = 'AND'
46      operator = [and_]
47  
48  class SearchOr(BinaryOperation):
49      name = 'OR'
50      operator = [or_]
51  
52  class SearchNot(UnaryOperation):
53      name = 'NOT'
54      operator = [not_]
55  
56  ############################### REUSABLE GRAMMARS ##############################
57  
58  AND = CaselessLiteral("and") | Literal('+')
59  OR  = CaselessLiteral("or")  | Literal('|')
60  NOT = CaselessLiteral("not") | Literal('!')
61  
62  searchTermMaster =  (
63      Word(alphas) | quotedString.copy().setParseAction( removeQuotes ) )
64  
65  ########################## THREAD SAFE PARSER FACTORY ##########################
66  
67  def like_parser(model, fields=[]):
68      class SearchTerm(object):
69          def __init__(self, tokens):
70              self.term = tokens[0]
71  
72          def express(self):
73              return or_ (
74                  *( getattr(model, field).like( like_escape(self.term),
75                                                 escape = LIKE_ESCAPE) 
76                     for field in fields )
77              )
78  
79          def __repr__(self):
80              return self.term
81      
82      searchTerm = searchTermMaster.copy().setParseAction(SearchTerm)
83  
84      searchExpr = operatorPrecedence( searchTerm,
85             [ (NOT, 1, opAssoc.RIGHT, SearchNot),
86               (AND, 2, opAssoc.LEFT,  SearchAnd),
87               (OR,  2, opAssoc.LEFT,  SearchOr) ] )
88  
89      return searchExpr + stringEnd
90  
91  ########################### SEARCH FIELDS LIKE HELPER ##########################
92  
93  def search_fields_like(s, model, fields):
94      if isinstance(fields, basestring): fields = [fields]
95      parser = like_parser(model, fields)
96      return parser.parseString(s)[0].express()
97  
98  ################################################################################
99

ListFilter Prototype

2009-04-03T03:59:00.000-07:00

Multiple space terminated regular expressions for list filtering.

The examples presented are somewhat hare-brained. a) Cause I'm hare-brained b) I'm really tired at the moment. Note the dramatic pauses :) Can.....you.....see....what....I'm.........do....ing You can imagine the usefulness though.

The filter is a prototype implementation of a filtering syntax for a QuickOpen Panel specifically designed for `quick` (try not to laugh) *multiple item* selection. The idea is to allow you to just keep typing in to refine your search. NOT this OR that. NOT that. `Open All In New Window` etc.

It is implemented using the editors actual text buffer API.

My editors current QuickOpen panel (while it does support multiple selection using ctrl-enter) works a lot like the FireFox url history search; space terminated tokens that each list item must match else will be filtered from the list. This works great for selecting one item but what if you want to open up multiple items at a time?

Plain regex searches are too unwieldy and lack the speed of entry desired.

The filtering is a regex extension of the current `all space terminated word chunks in any order`. It maintains most of the benefits and generally works as before for non regex characters. If you just need to type in some alphanumerics then it should be just as quick. It's essentially multiple regexes instead of multiple substring matches.

OPERATORS:

' ' AND operator
! NOT operator
| OR operator

jQuery = $;

2009-04-02T18:32:00.000-07:00

I have been making a foray into JavaScript recently for work and having had good experiences with `jQuery Lightbox` decided to use it for the basis of a job I was doing.

It is essentially a carpet gallery website with collections of colors (1:M). The designer wanted it to be `AJAX` ( a term that seems to have been hijacked to mean any page updated without slow browser refresh )

In the middle of the page is a large image. Above it are next/prev collection links and tiny `swatches` (thumbnails) containing links to each color for the current selection.

To either side of the image are next/prev color in collection links and upon mouseover a window will appear showing a magnified area following the cursor which is changed to a crosshair. The cursor will change via css styling to `cursor:wait` whilst waiting for the zoom image to load.

Upon changing color (via swatch, next/prev collection/color ) an animated gif will show while the chosen color's image is loading.

At first I was keeping a counter of the current colors position in the collections array of colors (clicking on the little thumbnails would take the title attribute, slug it and use that for the color, updating the current position index by with an $.inArray(color, colors) )

The filenames to load was, and are, a function of current collection and color.

The problem was having the state in an internal counter didn't really work for `open in new tab` or for sending links. "Hey check out this carpet... no the red one... Did you type in that url properly? Just paste it in."

I did some googling for `ajax urls` and stumbled upon a technique that sounded useful. That being polling the hash location for changes and then setting page state as a function of the hash. They said the ideal `polling` rate was 100ms. Sounded pretty hacky but at least the urls worked.

I searched for and found a plugin for jQuery that allows you to set event handlers for when the window hash changes. It uses polling but is responsive (42 ms) and works on IE 6.

I therefore just set a callback to update all the links and the image upon hash changing. I split the hash on '--' to find the current collection and color

The great thing about it is that the event is fired on page load so it goes through the `change color` routine. Updates all the links, shows the loading animated gif etc.

You can send links to people, open in next tab from any of the links etc

jQuery and its plugins made everything really straight forward. The only head scratching was getting the magnifying glass to work. None of the plugins worked out of the box for images that changed src. They worked mainly for a `static` gallery.

CTags 2: TDD

2009-04-02T17:55:00.000-07:00

This is part two of CTags. See part 1

Test Driven Development

Adherents religiously write tests first for *every* function they write.

Personally, I think a tonne of unit tests while prototyping is a waste of time for `simple` stuff with only one programmer working. It tends to feel like walking in mud. I prefer higher level tests that while they may not pinpoint the exact cause of failure won't double refactoring efforts. Especially when prototyping. Spend ages testing that `The Wrong Way` works? No thanks... Prototype, then rewrite with tests.

Sometimes though, `TDD` really is indispensable while exploring. Especially when working on stuff that is pushing the limits of your understanding. If I start encountering bugs I usually see it as a sign I need to start writing tests. Rather than using transient print statements to debug I'll write some unit tests.

The TagFile class below (now commented) is an example of when I found testing while prototyping invaluable.

  1  #################################### IMPORTS ###################################
  2  
  3  from __future__ import with_statement
  4  
  5  import os
  6  import bisect
  7  import mmap
  8  import unittest
  9  
 10  ################################### CONSTANTS ##################################
 11  
 12  
 13  """
 14  The tags in a `tags` file are listed one per line formatted as so:
 15  
 16      tag_name<TAB>file_name<TAB>ex_cmd;"<TAB>extension_fields
 17  
 18  """
 19  
 20  # symbolic constants for column indexes
 21  SYMBOL = 0
 22  FILENAME = 1
 23  
 24  ################################################################################
 25  
 26  class TagFile(object):
 27      def __init__(self, p, column):
 28          """ 
 29          
 30          Instantiate a new TagFile
 31  
 32          @p            path to `tags` file
 33          @column       which column to read
 34  
 35          """
 36  
 37          self.p = p
 38          self.column = column
 39  
 40      def __getitem__(self, index):
 41          "self.fh is the mmap opened by get"
 42          # Seek to a certain byte index
 43          self.fh.seek(index)
 44  
 45          # The position is likely to be halfway through a line so read up to
 46          # the first new line and throw away the `junk`
 47          self.fh.readline()
 48  
 49          # Note that it's actually returning the column from the line *after* the
 50          # line region containing the index.
 51          
 52          return self.fh.readline().split('\t')[self.column]
 53  
 54      def __len__(self):
 55          # bisect.bisect_left search must know how large the file is
 56          return os.stat(self.p).st_size
 57  
 58      def get(self, *tags):
 59          """
 60          
 61          Get all lines for one or more tags
 62          
 63          """
 64  
 65          with open(self.p, 'r+') as fh:
 66              # mmap is not needed but delivers performance increase
 67              self.fh = mmap.mmap(fh.fileno(), 0)
 68  
 69              for tag in tags:
 70                  # As __getitem__ returns the colum from the line region *after*
 71                  # that containing the index pt then bisect( alias of
 72                  # bisect_right ) will give the wrong index.
 73  
 74                  b4 = bisect.bisect_left(self, tag)
 75  
 76                  # Move the file to the position found
 77                  fh.seek(b4)
 78  
 79                  # Iterate over file. There may be more than one tag line to get
 80                  # per symbol/filename
 81                  for l in fh:
 82                      # Compare search vs line at column
 83                      comp = cmp(l.split('\t')[self.column], tag)
 84  
 85                      # This handles the case of being `left of left` due to 
 86                      # __getitem__ index being left of symbol it returns
 87                      # ie wait until catch up
 88                      if    comp == -1:  continue
 89                      # If line is greater then have moved on to next symbol
 90                      elif  comp:  break
 91  
 92                      # Found tag!
 93                      yield l
 94  
 95              # close mmap
 96              self.fh.close()
 97  
 98  ##################################### TESTS ####################################
 99  
100  class CTagsTest(unittest.TestCase):
101      def test_tags_files(self):
102          """
103  
104          This test basically iterates over each line in the tags file creating
105          a list of lines for each unique symbol it finds. It then compares this
106          list to that returned by the TagFile binary search. 
107  
108          """
109  
110          # Successfully passed test on 10MB+ tags file
111          # tags = r"C:\Python25\Lib\tags"
112          
113          tags = 'tags'
114          tag_file = TagFile(tags, SYMBOL)
115  
116          with open(tags, 'r') as fh:
117              latest = ''
118              lines  = []
119  
120              for l in fh:
121                  symbol = l.split('\t')[SYMBOL]
122  
123                  if symbol != latest:
124  
125                      if latest:
126                          tags = list(tag_file.get(latest))
127                          self.assertEqual(lines, tags)
128  
129                          lines = []
130  
131                      latest = symbol
132  
133                  lines += [l]
134  
135  if __name__ == '__main__':
136      unittest.main()
137  
138  ################################################################################

CTags 1: Captain Bisect And His Rag (c)Tag Crew

2009-03-29T20:21:00.000-07:00

CTags

Ctags generates an index (or tag) file of language objects found in source files
that allows these items to be quickly and easily located by a text editor or
other utility. A tag signifies a language object for which an index entry is
available (or, alternatively, the index entry created for that object).

Python has a philosophy of `Batteries Included` (and `Designer Straight Jacket`)
and its standard library has many useful modules to allow you to zoom out and
fly. Say what? What pixie dust have you have been snorting!?

The higher level you are the higher level you can go. If your stuck in the
trenches of detail it's hard to see emergent patterns to exploit which can help
you simplify code. Refactoring and simplifying is generally an iterative process
of simplifying, getting a higher level view and simplifying again.

Taking a bottom up approach in this helps as you know what building blocks you
need to create along the way. Don't want to be *too* simplistic. How can you do
things `top down` if you aren't `on top`?
Maybe you have flown over many times already and are just implementing an old
innovation from a time honoured map. That's exactly the spirit of prototyping.
Sending in the scouts to survey the terrain and report back.

However, those troops get mighty mutinous really quick if you don't take the
time to at least make a spot before you send em out like a swarm of jellyfish
(especially when they are being fired at)

Strategy? Fly over, send in the scouts and maintain communication.

What the hell is all that got to do with CTags? Not much I admit. But we does
love to babble.

Python std libraries are your `A Team` of special operatives that can do the
work you tell em without constant supervision and no need to worry about the
messy details. Go nuke PHP! "Yes Sir!"

One of these crackshot ninjas is Captain `bisect` with his special move
`bisect_left`.

He's a great leader because he's absolutely fastidious in keeping his company of
troops in perfect sorted order at all times.

Some of his duties involves training new recruits and he'll do a trick where
upon meeting them all for the first time he will get them to silently line up in
alphabetical order while he has his back turned. (we'll say alphabetical by
name, but well, these guys *are* macho army men so..)

>>> scum = "omewEDjyFapAdxslfhbgBcnCtkqzuvri"

>>> len(scum) == 32
True

>>> men = ''.join(sorted(scum))
>>> men
'ABCDEFabcdefghijklmnopqrstuvwxyz'

( Isn't it funny how the ones that try and make them selves seem big are in fact the smallest? )

" I bet I can find the position in the line of any one of you maggots after at most 5 guesses "

A, Yeah right, he thinks. "Find q sir"

bisect, "man 16 are you a lesser man than q?" k, "Yes"
bisect, "man 24 are you a lesser man than q?" s, "No"
bisect, "man 20 are you a lesser man than q?" o, "Yes"
bisect, "man 22 are you a lesser man than q?" q, "No" (smiles)
bisect, "man 21 are you a lesser man than q?" p, "Yes"

bisect, "man 22 you are q!"

troops, "Bravo Sir!"

( bisect is a a bit of a kookoo and uses 0 based indexing. Lucky the scum neverseem to get confused )

How does he do it!?

He starts in the middle (32 / 2 = 16) and compares what he was searching for with what he finds there. What he finds, k, is less than q so he knows he can rule out all other men before k as they too would be less than q.

This only worked for him because all the men are in order.

He then subdivides again. The man at position 16, k, is less than q so his area will start at
17 and extend in the opposite direction ending (as before) at 32 (the greatest man). He then looks for his next midpoint with an eye to ruling out another half of the remaining men.

( He repeats this simple process until his start point is no longer less than the
end point )

((17 + 32) // 2 = 24)
men[24] (s) is not lt q so his end point becomes 24

((17 + 24) // 2 = 20)
men[20] (o) is less than q so his start point becomes 21

((21 + 24) // 2 = 22)
men[22] (q) is not less than q :) so his end point becomes 22

((21 + 22 // 2 = 21))
men[21] (p) is less than q so his start point becomes 22

Start is 22 and end is 22 so he has a match!

With 64 men (double 32) he would only take 1 more guess; As each guess rules
out half and half is the inversion of double.

What about 128 (or some arbitrary number?) How many times can you half 128
before you get one (the right `one`).

Or in other words what to the power of two makes 128?

32 log 2 = 5
64 log 2 = 6
128 log 2 = 7

import math
math.log(128, 2)

Captain bisect's talent scales exceedingly well and in fact he's ready to put
forth his talent to whatever use come up with.

import bisect
bisect.bisect_left(some_sequence, search)

WOAH What a verbose explanation! Too much detail! Couldn't you just have said
use `bisect.bisect_left` to index left most occurence of any item in a sequence.

Yeah, that's kind of the point. Python is chock full of handy high level
abstractions you can trust to do the job without worrying about the details.
You are already zooming.

"Ok you admit your are babbling but what the heck is this got to do with
CTags?"

To paraphrase, Ctags generates an index (in sorted order) of symbols to
be quickly and easily located. Sounds like a job for Captain bisect. He is
actually made of this stern stuff called `C` that makes him faster than anything
you could genetically engineer in your laboratories.

Python also has this useful thing going called duck typing. bisect will work
on any class that exposes a __getitem__ method.

But what if you wanted Captain bisect to search a 50 MB ctags file? You can't
sub class a file object... can you? In any case each index would just return
a character wouldn't it?

A sneak preview of how to use bisect and mmap to binary search CTags `tags` files. Explanation to follow.

 1  #################################### IMPORTS ###################################
 2  
 3  from __future__ import with_statement
 4  
 5  import os
 6  import bisect
 7  import mmap
 8  import unittest
 9  
10  ################################### CONSTANTS ##################################
11  
12  # CSV Column in tag file
13  SYMBOL = 0
14  FILENAME = 1
15  
16  ################################################################################
17  
18  class TagFile(object):
19      def __init__(self, p, column):
20          self.p = p
21          self.column = column
22  
23      def __getitem__(self, index):
24          self.fh.seek(index)
25          self.fh.readline()
26          return self.fh.readline().split('\t')[self.column]
27  
28      def __len__(self):
29          return os.stat(self.p).st_size
30  
31      def get(self, *tags):
32          with open(self.p, 'r+') as fh:
33              self.fh = mmap.mmap(fh.fileno(), 0)
34  
35              for tag in tags:
36                  b4 = bisect.bisect_left(self, tag)
37                  fh.seek(b4)
38  
39                  for l in fh:
40                      comp = cmp(l.split('\t')[self.column], tag)
41  
42                      if    comp == -1:    continue
43                      elif  comp:          break
44  
45                      yield l
46  
47              self.fh.close()
48  
49  ##################################### TESTS ####################################
50  
51  class CTagsTest(unittest.TestCase):
52      def test_tags_files(self):
53          tags = r"tags"
54          tag_file = TagFile(tags, SYMBOL)
55  
56          with open(tags, 'r') as fh:
57              latest = ''
58              lines  = []
59  
60              for l in fh:
61                  symbol = l.split('\t')[SYMBOL]
62  
63                  if symbol != latest:
64  
65                      if latest:
66                          tags = list(tag_file.get(latest))
67                          self.assertEqual(lines, tags)
68  
69                          lines = []
70  
71                      latest = symbol
72  
73                  lines += [l]
74  
75  if __name__ == '__main__':
76      unittest.main()
77  
78  ################################################################################

GSOC OverView

2008-08-28T18:41:00.001-07:00

My GSOC project was all about testing for PyGame;

I wrote lots of tests; Almost every module in PyGame now has at least one test

Test modules can now be isolated in subprocesses; one segfault no longer brings down the whole test suite

Can now test for speed regressions; important for real time software such as games

PyGame Automated Build Page extended
- Shows / Collects more info
- Runs tests in subprocesses

Test Stubbing Utility: A Testing "Todo List"

Optional Interactive Tests / Test Tagging

For writing the tests I wrote a small utility that inspects the PyGame package and finds all the untested callables (functions, properties) and creates test stubs, including documentation for each so you don't have to leave the editor. The stubber knows which functions have already been tested by using a naming scheme for all of the tests. Essentially, "test_$callable__$comment", namespaced by having TestCase[s] per Class and a test module per module.

In this way I could create stubs for each module, essentially a TODO list, and cycle through all the modules looking for tests that were easy to write. The functions in PyGame are many and greatly varied, each requiring somewhat specialised knowledge to test. I wasn't able to write tests for all them but hopefully the test stubbing utility will help enable some testing sprints. I intend to develop a testing website where people can submit bugs/tests in the form of a unittest.

PyGame has a somewhat unique set of requirements compared to most python libraries in that most of the framework is actually written in C. C code when it goes awry can do some very strange things. We had a test runner running all of the tests in one single process so if one failed hard it would bring down the whole suite. This can be a bit of a pain so I developed a test runner that isolates each module in a subprocess.

Some of the tests in PyGame have requirements that make them unsuitable for running as part of the main test suite. For example some require a CDRom, a JoyStick, take way too long or need interaction with a human. With the test runner script I extended unittest with the ability to exclude certain tests by tags. The tags can be module, class or individual test level and are inheritable/ over-ridable.

Another extension to the test runner was the ability to randomize the run ordering of tests, so along with the test results the seed is printed out. If there are failures you can seed the randomizer with the failure inducing seed. We also wanted to be able to record the timings of each individual test so we could make comparisons between revisions / platforms. I again extended the test runner with that ability.

I worked with Brian Fisher to extend the PyGame automated build page to record the test results in a ZODB and utilize the new test runner to run tests in subprocesses. We will be able to use this information for detecting speed regressions amongst other things.

Johnny, Kick A Hole Right In The Sky

2008-08-23T08:25:00.000-07:00

Johnny, Kick a hole right in the sky! Won't some body testify? Poke a lion in it's eye!

I bought pygame-testify.net today, and set up a python/cgi based form that takes a zip and enumerates the results + adds the (safe evaled) test results dict to a ZODB.

I found a multi-part python snippet for POST[ing] of test results.

The test/build page is starting to come together.

I am using htpasswd for security.

todo_xxxxxxx

2008-08-02T16:22:00.000-07:00

I recently altered the "fail incomplete tests" mechanism we use in the pygame test runner. Before we were doing assertions on test_utils.test_not_implemented(). This would check a module level variable test_utils.fail_incomplete_tests, which we would set as desired depending on whether we wanted to fail incomplete tests.

This was a fairly non-invasive technique but as I was already hijacking the test loading mechanism for filtering tests by tags, I realized I could alter the TestLoader class to pick up tests starting with the prefix "todo_" as well as "test_". I would call TestCase.fail directly which would only run if picking up todo_ tests.

This of course meant altering all the stubs. I pondered briefly doing a mass search and replace, completely automating it but I don't really trust that for tests.

For the test stubs I have been including the documentation so it's really easy to walk through a test file writing tests without having to leave the editor. I was just using inspect.getdoc to get the __doc__ string.

It seems the documentation included in the .doc files is different to that contained in the __doc__ for each function. The __doc__ seems to be the function signature and a very brief, usually one sentence description. The .doc files contains a lot more detailed descriptions that can be very useful when writing tests.

I quickly added a docs_as_dict() function to makeref.py, then added it to the stub generator. The stub generator will add both the __doc__ and the .doc file documentation to each stub.

I went through semi-manually updating all the unfilled out stubs for each test file with the more complete docs and the new todo_xxxxx test naming. It took about an hour but I feel more confident than if I had just grep'd it.

Everything is pretty much now in place for the test site I wanted to create.

Test Timing
Test Tagging
Isolated Tests

import test.unittest as unittest

2008-07-18T21:14:00.000-07:00

I split the test runner further, now into three files, with all the monkey business in unittest_patch.

The patching is done by a patch() function taking an optparse options object as the solo argument, which drives the decisions behind which parts of unittest are patched.

With the features we wanted I had to override some methods in a quite drastic way. I even needed to override TestCase.run, a many many line method. The only way I could do this was to basically copy/paste, alter and monkey-patch in. This meant sometimes calling private members.

Unfortunately, the author of unittest had decided somewhere between python 2.4 and 2.5 that he would rename all the private members from the double underscore preceding __name_mangling convention to a single underscore _caution.

As my mentor said (or something like it), "using an underscore is a warning, that said member is an implementation detail not an interface".

What to do? We now include a 2.5 version of unittest in the test directory. Apparenly pygame has come full circle; it was included way back in the day before PyUnit was part of the standard library.

All of our individual test files, typically $module_test.py, all import an unpatched unittest and run unittest.main() to make the module "conveniently executable". Only when running the complete suite is unittest enhanced with extra functionality.

While I was in there tinkering with the internals, recording timings of individual tests I moved the redirect std(err|out) per module to per test. I then patched the TextTestRunner to dump stderr/stdout on error.

def printErrorList(self, flavour, errors):
    for test, err in ((e[0], e[1]) for e in errors):
        self.stream.writeln(self.separator1)
        self.stream.writeln("%s: %s" % (flavour, test))
        self.stream.writeln(self.separator2)
        self.stream.writeln("%s" % err)

        # DUMP REDIRECTED STDERR / STDOUT ON ERROR / FAILURE
        if self.show_redirected_on_errors:
            stderr, stdout = map(self.tests[test].get, ('stderr','stdout'))
            if stderr: self.stream.writeln("STDERR:\n%s" % stderr)
            if stdout: self.stream.writeln("STDOUT:\n%s" % stdout)

It would be relatively easy to add in support for show locals() etc.

Redesign

2008-07-15T18:12:00.000-07:00

I decided to (had to) redesign the test runner, this time cutting more directly to the root of matters, overriding select methods of unittest classes.

Before, in subprocess mode, I was calling the individual test modules, which would in turn run unittest.main() with all the attendant pains of cmd line options conflicting and output parsing. (we have to add profiling, exclusion by tags etc). One major design change I made was to unify the single / subprocess modes to use one test runner, (test_runner.py).

In it, along with a lot of utility functions, is defined a run_test() function. It takes a list of modules and an options object as arguments. It compiles a dictionary of the test results and on completion either returns the dict or in subprocess mode pretty prints it to stdout. (This is then eval'd for an all_results.update(result))

RESULTS_TEMPLATE = {
    'output'     :  '',         # unittest.TextTestRunner output
    'stderr'     :  '',         # stderr outpout 
    'stdout'     :  '',         # stdout output
    'num_tests'  :   0,         # taken directly from the unittest results object
    'failures'   :  [],         # ditto
    'errors'     :  [],         # ditto
}

In single process mode run_tests.py just imports from test_runner.py run_test() function and passes it the optparse options object and list of modules to search for tests.

Both run_tests.py and test_runner.py, share the same optparse cmd line parser options. In subprocess mode, run_tests.py calls test_runner.py with essentialy the same sys.argv it was initiated with. if __main__ it runs the run_test() function on a list of [args[0]]. Now all the extra functionality and cmd line parsing is all in one place.

There were quite a few extra little changes that have made it not perfect but a lot better. Adding exclution by tagging functionality took 10 minutes, most of the time being spent on picking a format.

|Tags:display|

Adding profiling decorators or whatever other functionality is desired will also be a lot easier now.

Comedy Of Errors

2008-07-10T18:09:00.000-07:00

** Build Page / Testing **
==========================

As reported earlier, in reaction to the crashing tests rendering the build page ineffective, I have been working on creating a script to isolate test modules in subprocesses. The approach I took, was to compile the results of each isolated test into the same form as the old test runner. A quick hack, or so I hoped.

I realised that subprocess out of the box has no cross platform non-blocking calls, so you can't timeout on hung tests. I had to find a recipe for this which unfortunately required win32 extensions. Not really a big deal but still time spent and dependencies.

So what we have is a test runner parsing the results of a unittest text report, meant for human consumption, which is then in turn parsed by the automated build pages regexes. This seems pretty ridiculous, especially as the form is not exactly machine friendly. I could have (should have?) hacked into the build page code and modularised the test parsing code there, sharing between the test runner and build page.

But then if you are going to do that why not just replace the TextTestRunner class with something completely customised for the job? Replace unittest bit by bit in an adhoc as-needed fashion? Slowly building a framework? I didn't want to. I'm not really supposed to be and that was the psychology in play.

Another? foolish design decision I made, based on a shallow visual aesthetic of less LOC, was to parse unittest results in a way that only worked when there was no "test noise". What do I mean by test noise? print statments left in source code. C extensions that don't respect sys.stderr, sys.stdout redirection/supression.

See below exhibit A, a specimen from a sunny day of testing.

...............
---------------------------------------------------------------------
Ran 15 tests in 1.234s

As the tests are running unittest prints to a stream, by default stderr, but it can be any file-like object of choice, either a dot an E or an F, mapping to pass, error or fail. I used a simple regex ^[.EF]*$ to find any "dots" in the return output. If there were any, I would take a slice from the length of the dots. From there I would take the first of a split at the "Ran xxx tests" boundary, defined as '%s\nRan' % (70 * '-'). In between the DOTS and the RAN_TEST_DIV (thus named) would lay the failures.

To piece it all together as if it was the output of one run I would "join the dots" and join the failures. Then at the end count the total length of DOTS (., E, F combined), E)rrors and F)ailures. Voila. Worked a charm.

What the hell was I thinking? The whole point of the exercise was to create a reliable test runner. I suppose I thought I was. I wrote a few tests for some spectacularly unimaginative cases. I compared output of single process mode and subprocess mode running some fake test suites, zero assertions, all passing, some failures, some errors. The subprocess mode was character for character perfect in its mime artistry. In fact it was for this easy, pull apart, bind together, compare automated testing that I did it in the first place.

All was simple and peaceful, until I finally got a linux test box working again. (my laptop fan died) I used ssh to log in and run the tests from my friends windows machine. Of course one of the tests that required initiating the display failed.

single process mode: 504 tests, FAIL (failures=1)
subprocess mode: 495 tests OK

What the hell was going on? With horror I realized what I had done. Something was wreaking havoc with the fragile little regex. On failure a huge amount of debugging output was put out by one of the SDL functions interupting the DOTS. I thought about rewriting it using some more substantive regular expressions. I tossed up between doing that and redirecting sys.(stdout|stderr) and passing a StringIO to unittest for test results. I figured by doing that I would be able to keep the comparison tests I had in place, and for that matter the same degree of mimicry. I opted for redirecting std(err|out). I imagined other uses for this at the same time, none all that compelling upon reflection and only useful if implemented in another manner. (only show stdout/stderr on failure of test, can leave print statement debugging in there, I did global redirection)

Of course to do that I needed to create a command line option for each individual test module to call from the "master" script in subprocess mode. Because unittest.main() is running with it's own getopt parsing, you can't just add an option and check sys.argv or use optparse. You have to do either and then clear those options from sys.argv which would otherwise cause unittest to error. So more fun hamfisting around with unittest. I realised that I would need to do that at some stage for profiling cmd line options so there was another push in that direction. All the time wondering whether I should just completely override the parseArgs method.

I replaced the test_utils.get_fail_incomplete_option();unittest.main() in each module with a test_utils.get_command_line_options(). unittest.main() always calls sys.exit() on completion of tests so I had to subclass it, overriding one of it's methods. I did this because after catching the unittest result stream to a StringIO, I would restore stderr and write the results to it.

I added in some test cases, print_stdout and print_stderr, comparing the results (I of course had to put a redirect mode onto single process mode for purposes of testing). Everything was OK again, until I ran it again on the the linux box through ssh.

495 tests OK. (should have been 504 with one failure)

Damn it! So it seems that some stderr, stdout is not redirected. I imagine it's mostly C extensions (or system calls) and the like that would do this but then that is pygame all over. Briefly I pondered printing results back out on stdout, and just PrayingTM that any such noise would always be stderr.

So what did I do? What any fool, already invested would do. I decided to markup the results, with lines like.

<!-- UNITTEST_RESULTS_START_HERE --!>

I created 3 sections using 2 divisors. The first is all the noise output, anything not respecting redirection. The second is the unittest results and the last is the multiplexed results, what you would see if running the script in a shell. I overrode the write method on a StringIO collecting unittest results and made it also write to (a previously redirected) stdout. using subprocess.Popen(...., stdout=subprocess.PIPE, stderr = subprocess.STDOUT) everything is muxed together. I then wrote a function that regex splits the 3, keeping the results for compiling DOTS. It's a long way from Kansas though isn't it Toto.

What a PITA? That's not even the half of it. I ended up having to rewrite all the command lines I was passing to subprocess.Popen from string template to lists so it would work cross platform. Also, the way subprocess multiplexes stderr and stdout when you use the same file object for both is inconsistent cross platform. What you would see is not neccessarily what you get. On windows it would suffice to just "print compiled_test_results", but on linux had there was need to print >> sys.stderr.

All in all, a lot of tipsy toeing around unittest. I really made a complete tangled webby mess of the whole job. A black comedy of errors. I'm not sure whether to remove the stderr/stdout redirection and replace the regexes with something less fragile. It's already been too much of a hole, sucking in time. I would have to update the run_tests__tests also.

What would I do differently looking back? What would I do if I had no constraints? Unfortunately, probably two very different questions.

** What I would do differently? **
==================================

This much I do know, the build page and the test runner script require intersecting functionality. They both parse the results of a unittest TextTestRunner output to gather statistics on test results. I could have modularised this parsing functionality, sharing between the two of them. This really begs the question though, why parse something designed for human consumption at all? Why not pass a customised test runner class into unittest?

Still there is the problem of communication across process boundaries, solved by using an asynchronous extension class of subprocess.Popen. Would you log the result of each processes output to a file using something like xml? Or maybe, pickling the results and then joining them back together? You could even have a client / server architecture, using sockets to transfer pickled test results as native python objects back to the server to piece together.

As well as the requirement for isolation of tests, we are wanting to add profiling functionality and tagging to split tests into different groups.

killer redux

2008-07-08T19:58:00.000-07:00

I was laboring under the bastard conception that when using subprocess.Popen(), shell=True is required for a subprocess executable to have access to the environment variables. Where the hell did I get that idea? Stupid unquestioned assumption that almost gave birth to a lasting bug.

For the test runner I was using system calls to taskkill or pskill for process controll under windows. The idea was to try executing each and if one was on the %PATH% the return code would not be one of err. If this was the case then the search was over and a Popen wrapper of (taskkill|pskill) would suffice as an os.kill().

This worked fine and dandy except that on windows98, there would be no error code if either of the task killers weren't on the path. It would define a useless os.kill.

Lenard, the windows maintainer of PyGame questioned why use a hacky wrapper of pskill or one of it's ilk, when if there was already a reliance on pywin32, why not use win32api.TerminateProcess?

That works fine but does not kill process trees, something I thought was a requirement due to using shell = True as a Popen constructor argument. Using shell = 1 calls cmd.exe etc which in turn calls the subprocess of choice.

Realizing that there was only need to kill one process, and that it would also avoid problems with differing return codes on older versions of windows, TerminateProcess was given the job.

Long live TerminateProcess.

dot points on build page extensions

2008-07-04T23:28:00.000-07:00

I have been thinking about making some extensions to the build page.

Raw Data

Keep raw_data to process at any time. No need to discount old data collected from buggy analysis.

Profiling

Use function wrappers, that log profiling of each test and multiple calls.

-p|--profile command line mode

Tests

Use subprocess mode by default for run_tests.py

Web interface for ticketing off tests

Build information

Post compiler version

Post complete Setup file

Post complete build output

Post complete test output

Python sys.path

Environment variables

As much as possible, unprocessed for archives

Machine information

Processor speed

CDRom availability

etc, etc.

Breaking up tests

Should the tests fail if a machine doesn't have a CD drive (assuming stubs were filled out) for example?

Should tests that require Numeric or NumPy fail if neither available?

There are some classes of tests that it seems to make sense to split apart from the main "base" group of tests. What should be the "base" group of tests to automate with the run_tests.py test runner?

What about tests that require human verification? For the build page a "base" group of tests should be specified.

What should be the requirements for machines sending results to the build page? Numeric, Numpy? win32 extensions on windows? A CD rom drive? 32 bit color display?

test_not_implemented()

2008-07-03T21:10:00.000-07:00

def test_get_arraytypes(self):

    # __doc__ (as of 2008-06-25) for pygame.sndarray.get_arraytypes:

      # pygame.sndarray.get_arraytypes (): return tuple
      # 
      # Gets the array system types currently supported.
      # 
      # Checks, which array system types are available and returns them as a
      # tuple of strings. The values of the tuple can be used directly in
      # the use_arraytype () method.
      # 
      # If no supported array system could be found, None will be returned.

    self.assert_(test_not_implemented())

test_not_implemented() will fail if any test suite is run with a "(-i|--incomplete)" command line option.

As mentioned in previous posts, I developed a unittest stub generator that will output stubs for any untested units. It is supported by a naming scheme for the tests. The stubber will inspect the xxxx_test.py modules and based upon the names of the unittest.TestCase's and their children test_xxxx methods will determine what is already tested.

For each public callable there is a corresponding test named test_$callable_name. Comments or descriptions will be appended to this separated by a double underscore.

test_quit__returns_None_if_not_already_init

What if there is a module.quit and a module.class.quit ? Each class has it's own TestCase (and thus namespace) named $classTypeTest. This is typically the case anyway with setUp()'s specific to the class tested.

def get_callables(obj, if_of = None, check_where_defined=False):
    publics = (getattr(obj, x) for x in dir(obj) if is_public(x))
    callables = (x for x in publics if callable(x) or isgetsetdescriptor(x))

    if check_where_defined:
        callables = (c for c in callables if ( 'pygame' in c.__module__ or
                    ('__builtin__' == c.__module__ and isclass(c)) )
                    and REAL_HOMES.get(c, 0) in (0, obj))

    if if_of:
        callables = (x for x in callables if if_of(x)) # isclass, ismethod etc

    return set(callables)

The script uses inspection to find all testables in pygame but there were a few complications, for example getter/setter properties and the fact that some objects need to be instantiated before inspection reveals their innards. Also, filtering out non-pygame callables and after that callables that appeared in more than one module.

eg pygame.rect.Rect led a double life as pygame.sprite.Rect. Just check the __module__ attribute ?

In [4]: pygame.sprite.Rect.__module__
Out[4]: 'pygame'

The workaround was to make a mapping of object to the place where it was defined. There were only 9 of these.

REAL_HOMES = {
    pygame.rect.Rect         : pygame.rect,
    pygame.mask.from_surface : pygame.mask,
    pygame.time.get_ticks    : pygame.time,
    .....

On some of the classes the __module__ attribute was __builtin__ so I needed put an exception for them in the filtering out of non pygame callables.

In [7]: pygame.cdrom.CDType.__module__
Out[7]: '__builtin__'

def module_stubs(module):
    stubs = {}
    all_callables = get_callables(module, check_where_defined = True) - IGNORES
    classes = set (
        c for c in all_callables if isclass(c) or c in MUST_INSTANTIATE
    )

    for class_ in classes:
        base_type = class_

        if class_ in MUST_INSTANTIATE:
            class_ = get_instance(class_)

        stubs.update (
            make_stubs(get_callables(class_) - IGNORES, module, base_type)
        )

    stubs.update(make_stubs(all_callables - classes, module))

    return stubs

The stubber finds all modules in the pygame package. For each module it uses inspection to create a set of all the callables minus those set in the IGNORE setting. This is here for any exceptions to the filtering and also for tests that have been grouped under one test name. These objects will not be stubbed.

IGNORES = set([

    pygame.rect.Rect.h,           pygame.rect.Rect.w,
    pygame.rect.Rect.x,           pygame.rect.Rect.y,

    pygame.color.Color.a,         pygame.color.Color.b,
    pygame.color.Color.g,         pygame.color.Color.r,

......

From that it creates a subset of "classes", the criteria being that for each element "inspect.isclass(element)" or that the element is in the manually set MUST_INSTANTIATE dict. This is a mapping of class to helper function, and instantiation args required to return an instance.

MUST_INSTANTIATE = {
    
    # BaseType / Helper               # (Instantiator / Args) / Callable

    pygame.cdrom.CDType            :  (pygame.cdrom.CD, (0,)),
    pygame.mixer.ChannelType       :  (pygame.mixer.Channel, (0,)),
    pygame.time.Clock              :  (pygame.time.Clock, ()),


..

}

Inspecting the xxxxType would reveal no methods, and they needed to be instantiated, but then the object returned contained no other attributes; one example being __name__ needed later for determing the test name. Therefore the xxxxType was sent to the stub generation function as the "parent class" for each callable that was gathered by inspecting the instantiation.

Any callables not in the "classes" set are assumed module level functions and a stub is created for each.

The test stubber is used from the command line:

$ gen_stubs.py --help
Usage:
$ gen_stubs.py ROOT

eg.

$ gen_stubs.py sprite.Sprite

def test_add(self):

    # Doc string for pygame.sprite.Sprite:

    ...


Options:
  -h, --help        show this help message and exit
  -l, --list        list callable names not stubs
  -t, --test_names  list test names not stubs

$ gen_stubs.py pygame -l
pygame.base.error.args,
pygame.bufferproxy.BufferProxy.length,
pygame.bufferproxy.BufferProxy.raw,
pygame.event.Event,
pygame.image.tostring,
pygame.joystick.Joystick,
pygame.key.get_repeat,
pygame.mask.Mask,
pygame.mixer.Channel,
pygame.movie.Movie,
pygame.overlay.overlay.display,
pygame.overlay.overlay.get_hardware,
pygame.overlay.overlay.set_location,
pygame.pixelarray.PixelArray.surface,
pygame.sprite.AbstractGroup.add,
pygame.sprite.AbstractGroup.add_internal,
pygame.sprite.AbstractGroup.clear,
pygame.sprite.AbstractGroup.copy,
pygame.sprite.AbstractGroup.draw,
pygame.sprite.AbstractGroup.empty,
pygame.sprite.AbstractGroup.has_internal,
pygame.sprite.AbstractGroup.remove,
pygame.sprite.AbstractGroup.remove_internal,
pygame.sprite.AbstractGroup.sprites,
pygame.sprite.AbstractGroup.update,
pygame.sprite.collide_rect,

Commas are appended for easy copy/paste into IGNORE list.

gen_stubs.py is an integral part of the plan to make it extremely easy for people to contribute to unittests. One man can only do so much.

subprocessed

2008-06-30T20:05:00.000-07:00

PyGame tests are structured in such a way that for each module in the pygame package (eg pygame.sprite, pygame.color) there is a test/xxxx_test.py file containing corresponding unittests. PyGame has an automated build page that shows build and test results for the latest svn version of PyGame on a variety of platforms and versions of python. It uses regular expressions to parse the results of the test runner script.

The test runner script compiles tests from each of the xxxx_test.py files and runs them in a single process. Advantage: speed, disadvantage: instability. PyGame uses a lot of c code, and where there is c code there is potential for strange errors.
As one example, there was a test for the ability to save OpenGl surfaces which would segfault on windows. This would stop the test runner half way through, leaving it's output in a form the automated build page could not decipher.

"Build Successful, Invalid Test Results"

Other issues with running all tests in one process is the need to restore a "fresh" state for tests that rely on it. Conflicts can cause the test runner script to crash completely. On the other hand, some obscure bugs have been uncovered due to them.

Besides writing tests for individual units I have recently been working on adding a subprocess mode to the python test runner script. It processes the output of each module's test script and outputs the results in the same form as the single process mode.

There is a library called subunit that uses os.fork() to run unittest suites in subprocesses, that seemed like it would have been a perfect candidate for the job. Unfortunately windows doesn't have the fork system call so it was not an option. Windows python does not even provide os.kill().

What good is running all tests in subprocesses if one of them hangs and python is using a blocking call to retrieve it's output?
As I was going to the trouble of making a subprocess mode, I realized I should deal with this possibility. Unfortunately the python subprocess module doesn't ship with async calls but I found a recipe on the ActiveState Python CookBook site.

On windows it relies on win32pipe and win32file from the pywin32 package. I worked around the lack of os.kill on windows by using sytem calls to "taskkill" or "pskill". If a wayward test suite running in a subprocess doesn't finish up in a specified allowance of time then it will be os.kill'd.

COMPLETE_FAILURE_TEMPLATE = """
======================================================================
ERROR: all_tests_for (%s.AllTestCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test\%s.py", line 1, in all_tests_for

subprocess completely failed with return code of %s

cmd: %s

return (abbrv):
%s

"""  # Leave that last empty line else build page regex won't match

Running each test suite in a subprocess is a huge performance hit. I think for the automated build page the performance hit won't really effect the experience as it's all running headlessly from cron jobs. Nevertheless, I added the ability to run subprocessed tests simultaneously in multiple threads. Also, the single process mode is still available as is running module specific tests suites.

I wrote some tests comparing the output of (single|sub)process modes running a group of fake test suites, some all OK, some with errors and failures.

$ run_tests__test.py
all_ok suite OK
failures1 suite OK

2/2 passes

-h for help

$ run_tests__test.py -h

-v, to output diffs even on success
-u, to output diffs of unnormalized tests

The standard library module difflib is very good, and extremely well documented.

Other than the obvious differences such as timing which are normalized before comparison, all is OK :)

Testing

2008-06-25T18:48:00.000-07:00

Just a quick note,

Still waiting on my fan, it's getting shipped in from Amurricah.

I wrote a run_tests_sub.py the other day that uses subprocess to run each xxxx_test.py in the trunk/test directory.

It will run with an optional threads paramater: -t num_threads

$ run_tests_sub.py -t 4

Apparently this runs faster on mult-core.

It should output results similar to run_tests.py though may need tweaking to get it run transparently in place of run_tests.py for the build page.

Speaking of the build page, Rene and I have had a few ideas for a combined build / test web app that collected builds and test statistics ( profiling / passes etc). Also, a means to distribute the writing of tests. Many hands make light work.

If it was possible to be assigned a stub of a test to fill out and then post it back painlessly we could quite quickly increase the coverage of our tests. If twenty people filled out 1 test a week, then over a month that would be 80 extra unit tests.

ATM there are "FAILED (failures=232)", unimplemented tests and possibly that many again that haven't been stubbed out waiting to be written.

$ run_tests.py -i

Will show tests that need fleshing out.

Aha!

2008-06-20T22:41:00.000-07:00

I realised why the change from CONSTANT = (expr) to CONSTANT = [expr] fixed the bug in the color_test.py

A generator expression is only good for one iteration and after that it will act as an empty sequence. I thought it would be a reusable lazily evaluated simily of a list comp. Turns out I was dead wrong.

I went over the stub generator recently and it's pretty much in it's finalized form as far as the naming scheme is concerned.

Can't wait to get my own computer back in action.

Fan

2008-06-18T03:26:00.000-07:00

Damn fan on my laptop packed it in. I will have to convince my friend to let me install linux on his windows box while I wait for a replacement. Can't get a windows build of development pygame at the moment due to failing tests... or can I? Temporarily disable the failing tests and let the build farm run?

What a pain in the arse.

Happenings

2008-06-12T21:05:00.000-07:00

I have been writing unittests using the naming scheme (see below) keeping to it as much as possible.

There have been a few modifications but that's fine as long as I am consistent. I haven't yet written the part of the test stub generator that filters from the generated tests any tests for units that have already been written. I am letting the writing of tests dictate the naming schemes evolution.

Will post some more thoughts on the naming scheme in days to come. Also thoughts on one to one test names.

Thoughts on speed of test suites, isolating "dangerous" tests that can crash the whole test suite.

Thumb Rules: Testing Generalities

2008-06-12T21:00:00.000-07:00

It's been quite an illuminating experience spending most of my time of late writing tests. A lot of the rules of thumb I have read about have proven to make good sense.

With the unittest framework especially, it seems to pay to keep to a minimum the number of assertions per test. Sometimes this increases the verbosity of your tests, something that python programmers seem to almost despise. People tend to optimize for elegance of code and to some extent performance. Test code should probably should be optimized to other priorities

Why does it pay though? The unittest framework stops any test dead in it's tracks on any error or assertion failure. If you have a whole batallion of assertions grouped under one test, on a failure you will only get information about one failure, and you miss out on a lot of context that would otherwise have been reported. Is there anything common to all these failures?

For example, there was a bug in some of the sprite collision tests (that intriguingly was not apparent on windows). The problem eluded me, but my brother discovered that it was due to testing the equivalence of lists of sprites, one of which was sourced from a dict, the order of which can not be guaranteed.

At the time there was a squadron of tests under one test, test_spritecollide.
( I restructured the assertions while renaming tests to fit the test stubber naming scheme. )

Running the test on linux, it would only report one failure when in fact the bug was repeated through about 4 - 5 assertions. Fix one and then the next assertion would fail.

Had they been structured in a way with less assertions per test, showing all the failures, it's possible a programmer of less ability like myself would have been able to solve the problem. Other programmers instantly.

This context makes things easier to hone in on the real problem. This brings me to another thought. unittests, what are they?

Unit tests; tests of units. Leaving aside the definition of unit, tests for what? One could say that they are testing for defined behaviour. Essentially then, you are testing for bugs, as if the unit is not working within defined behaviour, the unit is buggy.

What do you do when something is buggy? You debug of course. Tests then could or even should be debugging aids, especially useful when the tests are written before the actual units.

If you are not using tests for debugging then what? Some temporary scaffolding that with a tiny bit more energy could have been a test?

How to make the tests help in debugging without spending too much extra energy?

I'm not really sure on this, other than trying to keep assertions per test to a minimum. Another thing that may help is to not use anonymous expressions in the tests. If everything has been named, it's easier to use a debugger and get a glimpes of what's going on.

Tests should probably sacrifice compactness and abstraction for explicitness. There is probably a line to draw somewhere near repeating yourself too much (copy paste programming).

Sometimes it is easy to miss some bugs in your tests that give a false sense of everything being "OK".

An example of that is some tests for the Color type properties. I was looping over a generator expression that I defined for some test fixtures. All the tests were passing a OK. It turns out that for some reason ( I still don't know why ) in the scope of the tests, the expression was behaving as an empty sequence.

eg

for fixture in []:
assert something_about(fixture) == something

Nothing was ever asserted, the tests passed. I since changed the generator expression to a list comp and the tests are asserting themselves. ( expr ) => [ expr ]

blog.update(recent_events)

2008-05-27T02:18:00.000-07:00

I have resolved the failing test issues I had on Ubuntu.

It was a bug in testing the content equality of two lists; one was the return value of a function, the order of which could not be guaranteed. Odd that it was only an issue on Ubuntu, and presumably the linux platform in general. I commited a patch that fixed it.

At the moment I am working on creating a script, and supporting test structuring, to automate test stubbing for all units that are not tested.

This is so I can pick off the tests more methodically. Also, if it's possible to get a large group of people writing tests, just one at a time each week, a lot could very quickly get done. How to reduce the overhead so that is worthwhile and easy to be assigned one unit and write some tests?

The basic idea is to have a naming convention (possibly renaming existing tests) for tests that makes it easy to see if units are tested. A one 2 one mapping of callable to test.

test_$Class__$callable__$description

$Class             if a method
$callable          
$description       if any / optional

At the moment the script just creates test stubs for all "public" callables in the pygame package with that naming convention. It appends the callable's doc string as a comment.

def test_Surface__get_colorkey(self):
    """
    TODO: Test for unit, get_colorkey

    """

    # Docstring:

    # Surface.get_colorkey(): return RGB or None
    # Get the current transparent colorkey


    self.assert_(not_completed())

I am still fleshing out the idea

Compilation of Blues

2008-05-06T22:58:00.000-07:00

I was advised that I should get some experience compiling pygame. I use windows and thanks to Brian/Lenard it's a one click process to install the latest version of pygame. I knew that in the future I would be needing to run tests on Linux so I thought I may as well set it up now and compile pygame on that. I toyed briefly with the idea of setting up a dual boot. I have ran that setup in the past and it's a PITA so I opted for a virtual machine running Ubuntu 8.04 running on one of my virtual desktops. Virtual insanity.

After setting up the VM and installing Ubuntu I decided to have a crack at compiling everything from source and not using apt-get. This both for the experience and because I was under the mistaken impression that the svn HEAD version of pygame required newer versions of it dependencies than available through apt-get packaging.

(The wiki has compilation steps for Mac and Windows with links to source archives of pygame dependencies. A note on the wiki says "We should have a download with everything included. As well as patches for each one that we need". Sounds like a great idea to me. Maybe an svn repo? Is there much difference between the platforms?)

I downloaded the packages and began the quite tedious process of compiling everything.

............

I eventually had all the dependencies compiled and was glad to be finally able to begin compiling pygame

$
sudo python setup.py install

............

building 'pygame.scrap' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -D_REENTRANT -I/usr/X11R6/include -I/usr/local/include/SDL -I/usr/include/python2.5 -c src/scrap.c -o build/temp.linux-i686-2.5/src/scrap.o
In file included from src/scrap.c:59:
src/scrap_x11.c: In function ‘_convert_format’:
src/scrap_x11.c:77: error: ‘XA_PIXMAP’ undeclared (first use in this function)
src/scrap_x11.c:77: error: (Each undeclared identifier is reported only once
src/scrap_x11.c:77: error: for each function it appears in.)
src/scrap_x11.c:79: error: ‘XA_BITMAP’ undeclared (first use in this function)
src/scrap_x11.c: In function ‘_add_clip_data’:
    
............

My happiness was shortlived. I googled the problem and found someone back in 2005 having similiar problems trying to compile qt. It was something to do with not having the XFree86 development package. Turns out this package goes by the name libx11-dev for debian/ubuntu.

I installed it but the problem remained so I turned to #pygame for help. Someone helpful there, I can't recall who, said that it seemed I was missing some X11 header files and specifically a file called Xatom.h. Off in circles again for while looking for a missing package. I was pointed toward packages.ubuntu.com.

I searched for all packages containing Xatom.h. and..... aha! "x11proto-core-dev". That must be it!

$
sudo apt-get install x11proto-core-dev

It was already installed.

What now? I harassed one of my mentors, Rene, and he said that it could be a problem with pygame-trunk/src/scrap_x11.c or that it could possibly be because I got a non X version of SDL. "Damn.., I don't think I did"

He taught me about "sudo updatedb" and "locate X" with which I was able to confirm the existence of and location of Xatom.h on the system. He recommended that I use apt-get to get ready made packages of the dependencies and compile again.

I managed to replace all the local versions of the dependencies with apt-get ones but the first time round I missed deleting a lot of them. I have not that much experience with linux. The first time I recompiled it worked but upon running the unit tests I found that 5 of them were failing.

Rene, was quick to the rescue. "my guess is you're linking to your self installed sdl_image... try ldd `find . -name imageext.so`"

That confirmed it. I hunted down all the non apt managed packages and exterminated them from the system then compiled pygame again.

1 Unit test was still failing. IS still failing.

1  ======================================================================
2  FAIL: test_spritecollide (sprite_test.SpriteTest)
3  ----------------------------------------------------------------------
4  Traceback (most recent call last):
5    File "test/sprite_test.py", line 68, in test_spritecollide
6      self.assertEqual(sprite.spritecollide(s1, ag2, dokill = False, collided = sprite.collide_rect_ratio(20.0)),[s2,s3])
7  AssertionError: [<Sprite sprite(in 1 groups)>, <Sprite sprite(in 1 groups)>] != [<Sprite sprite(in 1 groups)>, <Sprite sprite(in 1 groups)>]

This test passes fine under Windows.

The hunt is on.

Journey of a Thousand Miles

2008-04-29T02:41:00.000-07:00

On the advice of my brother I have changed the name of the blog to something more generic so I can keep the blog once the winter has passed. You will be hearing from me soon. I want your ideas.

Winter Of Testing

2008-04-22T20:30:00.000-07:00

Wow, my "Google Summer of Code" application was accepted. My inbox has been flooded with congratulations and introductions from people from all over the world, who likewise were accepted in the program. Brazil, China, Russia to name just a few places. Hundreds of people excited about being sponsored to work on a project that interests them "over the summer". I live in Australia, and summer has just ended; no more morning swims, I will leave that to the diehards. Winter is here and it looks like I will spend it underneath a blanket at the keyboard.

For my GSOC project I will be extending the coverage of tests in pygame. With Py3K on the horizon and lots of other pygame ports in conception it's critical to get more tests in place. I will be extending the unittests and implementing some speed regression tests. Also, I will develop some interactive tests for things that are difficult/impossible/notWorthTheTime to test automatically.

My general strategy will be "breadth first", cycling through the modules. As time allows I will pinpoint target modules for intensive testing based on community feedback. I look forward to working with the PyGame community and welcome any ideas and advice.