In August Productions

I've been making some improvements to our use of Hudson recently that have been really helpful.

The first was starting to use a set of parameterized builds. These use the Parameterized Trigger plugin, which allows you to pass parameters to triggered additional jobs when a job finishes. So using these we make a job which checks out the latest source (which should just about always succeed) and then fire off a whole host of jobs running on all of our build hosts. It has a single "build now" submit form which takes a bzr branch location as the branch to build. With this all of our developers can check their tree against the build farm before submitting the branch for merge.

That's great, but as we got more and more build hosts, we had to set up a job on each of them - and then having 4 machines which were all amd64 running Ubuntu 9.10 didn't help - it ran on all of them.

Enter PlatformLabeler

Robert Collins wrote a plugin for Hudson which checks a build slave for OS details when it connects (architecture, OS and OS Release) and adds Labels to the slave based on the values it finds. With that - instead of targetting a job to say, hades.inaugust.com, I can target that job to x64_64-Mac-10.4.1. I can easily see both platform coverage, and I can take advantage of having multiple machines with the same config, since it'll run the job on only one of them sharing that label. 

The next problem with that setup is that there is not a tight binding between the parent and child jobs. You can see that the parent job triggered the child, but you can't really get a single snapshot of "how did this job work everywhere?" If you're doing this pre-merge request, it might be nice to get a URL you can refer to saying "hey! check it out, it worked"

Enter Matrix Builds.

Matrix Builds are a single job. They allow you to define how you run the job, and then you can select which hosts to run it on. You can also set up arbitrary sets of flags and hudson will build a grid. So in our case, I set up a job (shortened for brevity)

 ./config/autorun.sh

 ./configure

 make distcheck

Then I told hudson to build it both with and without the --with-debug flag to configure, and on a set of our build hosts. Now when I kick it off, I get a page with an overview of how that particular build went. Also, having one build description means I don't accidentally do something different on one host.

Matrix Builds also have a nice feature for a sentinel or canary build, which let you build first on a host where you're pretty darned sure it's going to build, and then build everywhere else. In our case, if it doesn't build on 64-bit Ubuntu 9.10, it's very unlikely to build anywhere, so we don't bother trying. 

I still use the Parameterized Trigger from the Matrix build to kick of jobs for things like Valgrind and Sysbench when the build completes cleanly. Those jobs themselves can be launched independently (if you only wanted to check how valgrind is treating your new branch)

Moving forward, I would love to migrate our main set of build branches to Matrix builds, but Matrix currently lacks support for launching an individual sub-build. If it doesn't show up soon, perhaps I'll add it... ah the joy of open source! 

See the Hudson link at http://hudson.drizzle.org/view/Drizzle-param/job/drizzle-param/ for more details and examples.

Any good piece of infrastructure software should be able to pop up little windows on your desktop. :)

Yesterday at LCA in Stewart's Hacking Drizzle talk, it occurred to me that error messages should pop up little windows on my desktop. So I wrote an error message plugin which uses libnotifymm to send Gnome pop-up window messages.

I don't always get to post screenshots, so here you go:

Friday was my last day working for Sun Microsystems, which means it was also the end of the job that started when I was hired by MySQL, Inc a little over four years ago. For those of you who didn't know me before that, it should be noted that the other than periods of time when I have started companies and worked for myself, the next longest I've ever been with one company was 10 months. I mention that as one of the few quantifiable metrics I could cite in describing what a wonderful experience it has been to work for MySQL and then Sun. The people I've worked with have been nothing short of stellar.

I will be continuing my work as a core Drizzle developer, so this blog post notwithstanding, I doubt most people will notice the change. The only real outward difference I can think of is that now that I'm not a Sun employee, I am free to comment on the merger and on what some people see as the need to save MySQL from Oracle. I don't know if I will, but at least now I can.

I would say I will miss everyone, but I imagine I will continue to see most people with about the same frequency as before. One of the wonderful things about the Open Source world is that changes in business and legal entities don't often have very much effect on the coding that we do day in and day out... so let's keep up the good work everybody! See you on IRC...

I've got some code in lp:drizzle/staging right now that's on its way (barring major catastrophes) to trunk. It's not code that does anything sexy as far as the actual running server is concerned. It's a code cleanup branch.

Anyway - short story being - everything from mysys and mystrings that is actually part of public APIs has been moved into drizzled/ proper. Everything else has been moved into drizzled/internal. None of the headers from drizzled/internal are installed... so none of the headers in drizzled/ should be using any of them. Combine this with the past week's removal of both server_includes.h and global.h, and we're getting pretty close to having fully consumable headers.

Which brings me to:

In doing this, I noticed a bunch of things that either need to be fixed, still need to be deleted, or need to be put behind a namespace so that including our headers doesn't strangely and unexpectedly b0rk someone's application. To that end, I've added a bunch of new low-hanging-fruit blueprints. As always, you can look at the full list on Launchpad but I'm including links to the new ones here.

New Drizzle Low Hanging Fruit:

http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-consolidate-charset-headers

http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-merge-decimal

http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-merge-error

http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-move-sql-alloc

http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-types-into-namespace

http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-mem-root-namespace

http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-internal-namespace

http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-remove-myisam-depends

http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-remove-my-files

http://blueprints.launchpad.net/drizzle/+spec/code-cleanup-remove-sql-list

Even the most casual of acquaintances knows that I really, really hate software that doesn't understand how we do things out here in the Open Source world. We have tools, really amazing ones, called ... "Package Managers" which allow us to "Install" software and describe the "Relationships" between elements. Apparently they can't seem to figure out how to deal with software dependencies and instead they opt for the "copy into tree and fork" method:

Google is forking existing FOSS code bits for Chromium like a rabbit makes babies: frequently, and usually, without much thought. Rather than leverage the existing APIs from upstream projects like icu, libjingle, and sqlite (just to name a few), they simply fork a point in time of that code and hack their API to shreds for chromium to use. This is akin to much of the Java methodology, which I can sum up as "I'd like to use this third-party code, but my application is too special to use it as is, so I covered it with Bedazzler Jewels and Neon Underlighting, then bury my blinged out copy in my application.". A fair amount of the upstream Chromium devs seem to have Java backgrounds, which may explain this behavior, but it does not excuse it. This behavior should be a last resort, not a first instinct. What should happen is this:

[google] hey, it sure would be nice if we could use sqlite in chromium for our local db needs.

[google] hmm, the sqlite API doesn't mesh 100% with how I'd like chromium to use it

[google] hey sqlite upstream, there are a few places where we'd like to see API improvements 
            so chromium can leverage it for our use-case

[sqlite_upstream] hey google, so cool that you want to use our code.

* sqlite_upstream looks at google's proposed changes to sqlite *

[sqlite_upstream] interesting, you could try using the foo_bar function for your camel launching
needs, but the rest of the changes seem okay

* sqlite_upstream commits changes to the source tree in commit rev 12345 *

[sqlite_upstream] Our next release will support that.

[google] yay! we'll tell people to either apply our patch or use commit rev 12345 or newer. 

As much as I appreciate Google's attempt at openness documented yesterday - and it REALLY is great and many steps in the right direction, it would be great if they'd also learn some of the process we already use out here. If they think it sucks and want to improve on it, then great - but start from what's there, don't just ignore it and hope we don't notice.