In August Productions

Change is in this air this summer in Drizzle-land. One of our GSoC students, Vijay Samuel, has been hard at work replacing the options processing system we inherited from MySQL, my_getopt, with one based on boost::program_options. We've been merging his work in to trunk for a while now, and he's made really excellent progress, so it's probably about time to point out how the new system will be different from the MySQL one. There are three main changes afoot here, and I'm actually pretty pleased with all three of them:

1. Plugin Option Prefixing
2. Dashes v. Underscores
3. Config Files

A few of these changes are still in the middle of their transition, so I'm just going to describe the finished system, but we've essentially got all of the client programs and most of the plugins done at this point. 

Plugin Option Prefixing

In the old system, when a module registered a new option or variable, we would concatenate the module's name on to the front of the option name with a "_". So, for instance, the mysql_protocol module has an option for setting the port, which wound up being named "mysql_protocol_port". In the new system, boost actually understands grouping and namespacing of options, and we decided to take advantage of that by mapping a plugin module to a group. This means that the concat character is now a "." (that's a dot if your eyesight is bad)

Dashes v. Underscores

In addition to that, boost doesn't know anything about the long-standing MySQL tradition of going out of the way to allow you to do things as many ways as humanly possible. To Boost, the strings "mysql_protocol" and "mysql-protocol" are, well, different. We considered writing a parser plugin for it to do normalization for us, but then we made the other decision.

Dashes and Underscores are different.

Now this may royal screw some people who may have enjoyed making art by listing an option like "auto-increment_increment" before, and to those people I'm sorry, because I really do like injecting art into day to day life. Nonetheless, from this day forth let the decree be heard...

Options have all been normalized to use no underscores and to use only dashes.

On the off chance you can't follow the logic there, that means that the command line option for setting the MySQL protocol port in Drizzle is now: --mysql-protocol.port=3306

Config File Changes

But wait, there's more...

First of all, we changed from having one master config file for everything with sections in it for different programs to having one config file per program. drizzled will read "/etc/drizzle/drizzled.cnf". The command line clients will also read a few more, since you might want to repeat yourself - so drizzleddump will read ~/.conf/drizzle/drizzledump.cnf, ~/.conf/drizzle/client.cnf, /etc/drizzle/drizzledump.cnf and /etc/drizzle/client.cnf, with files listed first in that list overriding values in files later in the list. (In case you're wondering about the ~/.config part, it's part of the freedesktop.org Base Directory Specification. Since we didn't have a system for this before, it seemed sensible enough to just follow the standard for user/client utilities here.

The files are still mostly ini-style files, except they do not require a base [section] header. So just stick "datadir=/var/lib/drizzle" into /etc/drizzle/drizzled.cnf and you're off to the races. However - remember that grouping thing I mentioned before- in the boost world, section headings translate to option prefixes joined by, you guessed it, a dot. This means you can do:

datadir=/var/lib/drizzle 

[innodb]

buffer-pool-size=100G

log-file-size=1M 

Launchpad has two facilities for filing tasks that need to be done in a source tree, Bugs and Blueprints. I've spoken to folks who think the two should be collapsed in to one thing, and folks who think they need to remain separate. I'm in the latter camp, so I thought I'd weigh in real quick with a quick explanation of what I think the difference between the two is. Whether they are fundamentally the same object in the backend object model is irrelevant to me - I'm talking about developer workflow and thought.

In my view, it's really simple:

• A Bug is a record of something that needs to be done - the what
• A Blueprint is a record of how something will be done - the how

If you accept that premise, then you can have a task that has both a bug and a blueprint associated with it. Or you could have a bug with no blueprint (sometimes stating the problem implies the solution) It makes less sense to have a blueprint without a bug - but we do this as a shortcut often times because launchpad doesn't have an automatic 'create a bug for this' feature for blueprints. On the other hand, many of the blueprints the Ubuntu project works with during UDS planning aren't really things with bugs - unless you consider them all to be hows of solving bug#1. Or perhaps there is a missing or merely implied definition of the what for many of them, which might mean that the discussion of how meanders for a little bit.

I think the current structure of blueprints backs me up here. It is intended that a blueprint have a wiki page (implying that perhaps the information might be longer in form) and blueprints have a wonderful dependency graphing feature - which is quite useful for project planning ... "to accomplish this (bug#1) I'm going to do this (blueprint#1) which depends on these other things being done first (blueprint#2 and blueprint#3)"

Additionally, blueprints having an associated wiki page implies that anything with a blueprint is something that should turn in to documentation. If you're writing up a description of how you're going to do something, then perhaps that means when you're done you've written a description of how or why you actually did something, which is sort of what internals documentation is, right?

Now - can things be better with blueprints? Of course. There was discussion at the last UDS of having blueprints have a merge-request style discussion/approval process, which I would whole-heartedly approve of. Additionally, the wiki element could be baked in to the blueprint, and bzr branchable.

But I strongly believe that both serve a strong and fundamentally different purpose.

Launchpad is a wonderful and powerful tool. With that comes a variety of different ways different features can be used in the context of a project. I thought I'd take a moment to write down how we're using bugs and when we use what statuses. There are two goals here:

• We could do better at this ourselves
• Maybe it's useful for someone else who is getting themselves ramped up on Launchpad.

Step one is reporting a bug. This may go without saying, but at some point someone will file a bug, and we'll need to notice that - so we have launchpad bugs set up to mail everyone in our main drizzle team any time a bug is reported. This ensures that they don't go unnoticed.

When a bug is first reported, it starts life in the status "New" and with an importance of "undecided".

The next state is "Confirmed". In a perfect world, this bug achieves this state when someone other than the bug reporter has independently been able to reproduce the reported problem.

"Triaged" comes when the bug has been assigned an importance, someone to work on it, and an initial target for completion. Often Confirmed will get skipped and it'll go straight to Triaged, actually, because often times the person confirming the bug is also assigning to to someone.

A quick digression on targetting... in Drizzle we create a new Series for each planned release - such as cherry, dexter or elliot, and then we create bi-weekly milestones to track our bi-weekly tarball drops. So a bug can be targeted to a series and a milestone. The milestone may (and often does) move, depending on whether or not we were successful in getting it done. Once it's been targeting to a series though, it will forever have some status for that series. So if a bug starts in cherry and it doesn't get fixed and we release cherry and move on to focusing on dexter, we don't remove the bug from cherry - the bug is still there in the cherry release after all. If we were doing maintenance releases on things (which we're planning to do once dexter is out the door) we might leave the bug open in its cherry status until a fix has been made in trunk and backported to cherry. Or, since we're not currently doing that for cherry, we'll just set the bug to "Won't Fix" for cherry and track its status in dexter. 

This leads us to the next two statuses... Invalid and Won't Fix. It's possible that the bug, as reported, cannot be reproduced. In that case, instead of setting the status to "Confirmed" - we'll set it to Invalid. Similarly, if the bug can be reproduced, but we think the behavior is valid, we'll set it to "Won't Fix". We have not started to use the new "Opinion" status yet.

"In progress" means that the developer assigned to the bug has actually started work on the bug. This one gets missed the most, because it requires website interaction from the developer, which is usually not what they're thinking about doing when they're coding.

"Fix Committed" and "Fix Released" are the ones most up to interpretation in general. In Drizzle, we use them to mean "A Fix has been committed to some branch and pushed to Launchpad" and "The fix has been pushed to trunk" respectively. I would personally love a third status "Fix Merged" which I could set it to when we set "Fix Released" now, and then use Fix Released for when the fix has hit a tarball... but I'll live.

To sum up:

• New - reported and not looked at
• Confirmed - someone has reproduced it
• Invalid - could not be reproduced
• Triaged - has been assigned an importance and a release target
• Won't Fix - reproducible but no intention to change the behavior
• In Progress - a developer is actively working on it
• Fix Committed - a developer has pushed a branch containing a fix for this bug to Launchpad
• Fix Released - the branch containing this fix has been merged into trunk

While sitting around with Stewart, Eric, Max and Beer at the MySQL Conference and Expo, Stewart thought it would be funny if someone would do a graph showing the trend of Google searches comparing NoSQL and Your Mom. Always wanting to make Stewart laugh, I ran over to Google Trends to see if it could make a graph.

NoSQL vs. Your Mom

Given all the hype these days, you might think that Your Mom would stand no chance against the Juggernaut of NoSQL. But I was quite surprised to see that Your Mom really stuck it to NoSQL. Giving NoSQL some credit, it is making some progress.

Surely then it's a bogus test. I'm graphing tech jargon against a phrase in the common vernacular. It must be the case, given that MySQL is being replaced with NoSQL everywhere, surely adding MySQL in to the trends mix would just make me a sad little Fail Whale.

NoSQL vs. Your Mom vs. MySQL

Oops. MySQL still seems to be considerably more popular than Your Mom. Perhaps suppositions made while sitting in the Hyatt aren't always correct.