Recently in Programming Category

It’s not uncommon that I’ll want to remove initial spaces and trailing spaces when I’m parsing user input.

In the past, I’ve always done the following:

$text =~ s/^\s+//;  # Remove initial whitespace
$text =~ s/\s+$//;  # Remove trailing whitespace

As it turns out, though, these two lines can be combined into one if you add the /g flag:

# Remove initial or trailing whitespace repeatedly
$text =~ s/^\s+|\s+$//g;

For whatever reason, that didn’t occur to me until reading through someone else’s code where it was used.

If you need to do this sort of thing as well, and it hasn’t occurred to you, either, here you go.

This is a nice command. It’s simple. It works. And it puts out fires.

When you release an update and start getting automated error E-Mails shortly thereafter, it’s so much nicer to be able to say “go back to this revision, which I know worked” than to have to find the bug, fix the bug, and release a new revision while hoping that you didn’t just break something else since you’re doing it in a hurry before too many other people notice.

I used to do the latter, and am now trying to get into the habit of doing the former. This has the added benefit that I have more time to write the test case that should have caught the bug in the first case, so we’ll never see that particular bug again.

There’s a race condition in my application server that can cause a particular type of database query to fail. Perl’s Class::DBI module has a findorcreate function that does a SELECT query to see if a record exists, and then an INSERT query if it doesn’t. If some other application does an INSERT between those two queries, the first application’s INSERT will fail.

In my application, there’s a gap between those two queries of one tenth of a millisecond, or 0.0001 seconds. That’s a small window.

And yet, code has snuck through that window six times in the past three days, resulting in a “duplicate key value violates unique constraint” error. What makes this even more incredible is that this race condition only pops up if there has been another race condition failure earlier in the process.

Fixing the problem can be done in any (or all) of three ways:

1) Handle the database error. The records are identical, so in this case I can just quietly move on if the race condition strikes.

2) Use locks. Only let one instance work on this queue at a given time.

3) Add a random delay. This error is happening because three or four processes are all getting fired at once, and they’re all hitting the database at the exact same time, resulting in concurrency problems. Adding a randomized delay of 0-0.2ms would be enough to work around the problem, and I can implement this just by randomizing the order of operations, so it won’t affect performance.

I’ll probably implement all three, just to have plenty of assurance that it’s fixed, but I’m still incredulous that it’s needed at all with the current amount of traffic.

Don’t cache something if it takes longer to retrieve it from the cache than it takes to generate it in the first place.

Corollary: Improving performance on the cache mechanism allows more stuff to be cached effectively.

This was discovered while attempting to speed up the work list and order detail pages. Fortunately, adding an index in a particular place in the database was enough to get the cache retrieval and validation to be well under a millisecond.

That, combined with roughly eight hours of tweaking code and database calls, means that an average order detail page request now only requires 182 database calls, down from 689 (74% reduction), and is generated (in most cases) in well under a second, even without touching the cache.

With the cache, it’s generated in roughly a tenth of a second (with about ten database calls for the entire request), though I still have some work to do, since it’s taking the browser between a quarter and half a second to render.

The work list load time is just under a second now, with some selective caching, down from about 2-3 seconds, using a test set of 40 active mailings. That’s fast enough for now. Database calls also dropped from 789 to 402, which is a nice bonus. At 100 active mailings, it slows down to about 1.5 seconds with the selective caching (down from about 10 beforehand), so I’ll probably need to do some more performance work once we start seeing that on a regular basis.

Update a table according to the results of another query (slightly simplified):

UPDATE letters AS final
   SET preview_file_id = outer.preview_file_id
  FROM (SELECT letter_id,
               (SELECT preview_file_id
                  FROM inserts AS i
                 WHERE i.insert_id = inner.insert_id) AS preview_file_id
          FROM (SELECT letter_id,
                       MIN(insert_id) AS insert_id
                  FROM letters AS l
                  JOIN letter_insert_ref USING (letter_id)
                  JOIN inserts AS inner_i USING (insert_id)
                 WHERE l.preview_file_id IS NULL
                   AND inner_i.preview_file_id IS NOT NULL
                   AND l.submitted > NOW() - '1 month'::INTERVAL
              GROUP BY letter_id) AS inner) AS outer
 WHERE final.letter_id = outer.letter_id;

Or, in something resembling English, “In the letters table, set the preview file according to the following list of recent letters and the associated preview files for the first insert of each of those letters.”

There might be a way to simplify the two sub-queries into just one using grouping, ordering, and limiting, but this worked well enough for a one-off query.

Background: I thought I no longer needed the previewfileid field in letters, so I deprecated it recently, and stopped setting it. The API layer had a workaround, but there’s still a program that uses a database view that needs the previewfileid to be set in letters, so I needed to revert that change and back-fill the previewfileids that didn’t get set. UPDATE FROM to the rescue!

(Also, if anyone has any layout suggestions on how to make complex SQL more readable, I’m all ears. I’ve been center-aligning to the keyword for years just because I haven’t found anything better.)

It’s always satisfying when you’re getting ready to add a new feature to your program, hunker down to do it, and then realize that you can accomplish it by adding one character in the right place.

:-)

It’s admittedly not a major feature (changing the way that a CamelCase conversion handles all-caps acronyms), but I was expecting it to take several lines of not-very-pretty code.

I think this is going to be my last reflection post on OSCON. It’s getting late (remember, I wrote all of these back on Friday after the last session was over), and I’d like to spend some more time thinking down other paths before I call it a night.

The presentations this year were on the whole very good. Outside of the one tutorial I mentioned at the end of the last post, the baseline set by the presenters was a testament to O’Reilly’s ability to find good and engaging speakers, and help them prepare for their sessions.

Both Damian Conway and Paul Fenwick are masters of integrating their visuals with what they’re saying. The slides are independent but connected, with visual and speech each playing off the other, and the presentation wouldn’t be nearly as interesting (or hilarious) if the visuals weren’t present.

What’s important to note, though, is that it takes a tremendous amount of time to develop those slides and rehearse the timing such that you don’t even notice that they probably went through a couple hundred slides (including transitions) in 30-45 minutes. It helps to be able to give the same talk multiple times, but the up-front preparation is still the same.

In contrast, no fewer than three presenters (including one three-hour tutorial presenter) commented that they had written their slides in the last 12 hours. That’s not setting yourself up for a win.

They also knew their content. Some presenters had awkward silences as they were looking through their notes, reminding themselves of what they were supposed to say next. These two had their share of glitches, but they were much better at recovering from them.

Oh, speaking of glitches. The A/V company had some decent equipment for the sessions (not the keynotes — the quality there was pretty bad), but their wireless mics have a really nasty problem — they fail full-open, rather than silently. In other words, if the transmitter stops working due to a dead battery, the receiver decides that the appropriate thing to do is blast white noise as loud as possible. Sennheiser wireless mics, on the other hand, as well as the mics of any other manufacturer with any sense, fail silently — they send a known signal along with the audio transmission, and if that signal disappears, it mutes the channel. Fortunately, I was hanging out near the front, outside the path of the speakers, so I didn’t experience that pain like most of the rest of the room did.

Also on the topic of glitches. This seems pretty basic, but you should really know how to use your presentation software before you walk into the room to give your presentation. That fellow was really nice about it and appreciative when I showed him how to use Keynote, but it does show a certain lack of preparation and professionalism. Maybe he had already been rescued from another disaster by having his presentation imported into Keynote from some other software that stopped working at the last minute. Dunno.

Slide transitions were well done throughout the conference, which was nice. There were very few animated transitions, and they were usually an integral part of the presentation. One person used fades. The rest just used straight cuts, which is the right thing to do. That was pleasant.

They were also very good on the whole about font size. I did make a point of sitting near the front (whenever I didn’t, a very large person would invariably sit directly in front of me and do a lot of fidgeting throughout the talk), but it rarely struck me that people in the back wouldn’t be able to read the text.

Most people also didn’t read their slides. That was really nice. I’m trying a demo of a popular Software-as-a-Service CRM package that prides itself on its “No Software” mantra (meaning no software that needs to be installed), and their tutorials are mind-numbingly boring. Not only do they read all of their slides word for word, they also read the titles out loud, and talk at a rate of about 30 words per minute.

So, here at OSCON, even the weaker presentations were pretty good, at least for those that I attended. The only real challenge (outside of that one bad tutorial) were the occasional language comprehensibility issue, since the conference draws from all over the world. Oh, and one speaker let his session get hijacked by someone in the audience, which was also noticed by at least two other people who mentioned it in blog posts after the fact. That was unfortunate. But, on the whole, quite pleasant.

And with that, I’ll wrap up my reflections on OSCON, and go back to thinking about how best to run and grow my business.

If you like programming in a non-Windows-centric environment, I can’t recommend it enough. (It’s probably of less interest to someone who spends most of their time in .NET or C++, unless you’re also drawn to open source.) If programming is your day-job, particularly if you use a scripting language like Perl, PHP, Python, or Ruby, talk your boss into sending you here — the productivity boost that it will give you will be well worth the company’s investment, even in the short term.

Perl is an insane language. I love it.

I’m guessing that there are very few languages that can be made to interpret rod logic, as shown by Damian Conway during the opening night’s keynote (or Latin, or Klingon, to highlight other talks he’s done), nor are there all that many languages that let you write positronic variables that let you return results before you’ve calculated them (same talk).

But even if you discount the things you can do with filters, how many languages let you do something like this:

*{";\n"} = sub { print something }

In case you’re not familiar with some of the lesser used Perl syntax rules, that says “make a function called ‘semicolon newline’ and have it print ‘something’ whenever it’s called.”

That will probably result in a “so what” response from the non-techies reading this, but it should cause a certain tightening of the stomach and a feeling of low-level despair to any programmers reading this, possibly with a little jaw-dropping, once they realize what it means. (Hint: What is at the end of every statement in most languages?) And maybe some anti-Perl flaming for even allowing such a thing (don’t worry, it gets worse — for the C/C++ programmers out there, you can also name a function “\0”). But you can write Fortran in any language.

Back in 2004, I attended a tutorial called Perl Best Practices. This year, I attended Perl Worst Practices. Damian C. commented that we must be the smartest, cleverest people at the conference to have convinced our bosses to let us attend a three-hour tutorial with that title. I’m not sure what that says for me. Other talks included Perl Security and The Twilight Perl (showing that things that should be syntactically impossible really aren’t).

What’s neat about Perl’s insanity is that it let’s you do practically whatever you want. Even if you shouldn’t. Even if 99.999% of the time it’s a monumentally dumb idea. And that allows best practices to be developed and codified over time, rather than being limitations imposed by the language (there is that 0.001% where it’s the perfect solution, and saves you days of writing workarounds). This is increasingly becoming a defining point of the Perl community, once you get away from the people who treat Perl like stereotypical PHP.

Learning some of these crazy tricks, while hopefully not the sort of thing you’d ever use in production, gives you a better understanding of the language. If nothing else, that can be really helpful for debugging, or if you ever have to maintain someone else’s code (or code that you’ve written more than six months ago).

Because, once you’ve spent three hours going through SelfGOL statement by statement, there’s probably not much that an inexperienced or undisciplined coder can do to scare you.

Outside of the Perl world, one of my goals for the conference was to get a better understanding of Ruby, since it’s getting a lot of attention. Well, I tried to be open-minded, but even the presenter reinforced that it’s still a relatively new and untried language that’s going through a fair bit of change as it’s maturing, and the syntax didn’t seem all that better when considering code to line noise.

The “fair bit of change” description is especially true for Ruby’s frameworks. Not that Perl doesn’t have its own problems with frameworks (it does, and I may write some thoughts on that at some point, since they’re mostly not safe to use, either), but Rails is probably the main selling point that Ruby has, to the point of being synonymous to a lot of people, and it’s still very much a moving target, without enough of an emphasis on backwards compatibility.

That makes it a very bad choice for any program that you want to stick around for a while, unless you’re willing to invest a lot of your energy in keeping up with the changes to your underlying framework, rather than enhancing your own code (that’s my big problem with Drupal as well, despite my really wanting to like it).

But lest I be accused of having a bias against anything that isn’t Perl, I did come out of the conference with an interest in learning more about the Mozilla framework, along with a renewed desire to help in at least QA with that project, with the hope of eventually getting to know it well enough to write some client software using its tools. And I went to a good session that did an introductory overview of C, since that’s a definite area of weakness in my skill set at the moment. I probably should have attended a Python session as well, but there are only so many timeslots available. Maybe I can set that as a goal for next time.

All in all, I think I was only in one session that was particularly bad. Otherwise, there was a huge range of quality, but I didn’t find myself wishing I had gone to a different session instead. So that was definitely a win.

More on speakers and presentation styles next time.

Part 1 was a generic overview of my experience at OSCON this year. In short, I had been looking forward to this conference for about three years, and I wasn’t disappointed.

An event like this would probably be an anthropologist’s or sociologist’s dream study. Get over a thousand mostly highly focused, technical people, 80%+ of whom are introverted, statistically speaking, all with a fairly narrow similar interest, and put them all in one place to see what happens. It’s a lot of fun.

What happens is that a large number of them open up. They’re finally among people who understand them. They have conversations with complete strangers, almost like they were extroverted. (It helps that there are some extroverted people to act as catalysts.) They have that common understanding with these people that they lack with “normal” people, defined as roughly 99% of the rest of the world.

If you think I might be exaggerating, consider this — how many conferences have a “People” track, wherein many of the sessions in that track are related to how to get along with and interact with other people? (And some of them, based on overheard feedback, included practices that many would consider pretty fundamental, like, oh, say, the importance of showering.) That’s how bad geeks can be, in general.

One of the better talks that I did end up attending in that track (while speaker-following) had as one of its titles “Hacking Wetware.” For the uninitiated, “wetware” == “humans”. Oh, and hacking can be more or less defined as “getting to understand at a fundamental level,” not as “breaking in and destroying” or “doing evil things”.

Among other things, including using The Sims as its overarching point of reference, it featured the gem of explaining the stereotypical greeting using the TCP three-way handshake:

• Hi, how are you doing? (SYN)
• Good (ACK), and you? (SYN)
• Good, thanks for asking. (ACK)

It’s great because it lines up so perfectly with the point of the greeting — you really don’t care how people are doing, you’re just establishing communication. I admit, I poke fun at people rather often by either changing or abbreviating line two, and watching them completely miss it. Though, the best one I witnessed was actually done by Christine at a restaurant, when she answered “Wet” (it was raining outside), to which the hapless greeter replied “I’m so glad to hear it. I’m fine.”

Anyway, back to geek-watching. My flight to OSCON was a somewhat poorly thought-through one-stop flight from Manchester to Portland with a layover in Philadelphia. Timing-wise, it worked well, but I hadn’t thought about the fact that Philadelphia is further from Portland than Manchester, which made for a really long second flight.

While in Philadelphia, it occurred to me that there was a decent chance that I might be able to spot other people who would be going to OSCON. So the game became how to spot them. (I did this last year for the National Postal Forum. There was a whole group of postal service employees on my flight.)

An observation that I made some time ago with regard to Christian conferences is that you can usually tell by what t-shirts people are wearing. Christians know that they’re not going to face much of any persecution while at such a conference, so they tend to wear all their religious stuff there. (This isn’t just a religious thing — you wouldn’t walk into a bar in Boston wearing a Yankees cap and jersey without expecting some persecution, and possibly higher prices.)

So, one person made it easy. He was wearing a t-shirt that asked “What’s your uptime?” Easy tell.

Looking at another person, I got the sense that he was a geek, but initially wasn’t sure. Right demeanor, right dress, right luggage, but nothing definite. It wasn’t until we both got on the same train to get to our (same) hotel that I noticed the subtle giveaway — he had a glider from Conway’s Game of Life sewn on his messenger bag. There’s no mistaking that, but only if you’re “in”. Turns out he’s a Perl geek, too, based on the sessions we were both in.

Back to sociology. The other thing that was amusing (and which I practiced a fair bit) was that you can completely ignore people and it’s perfectly fine. As was noted while the speaker was talking about the three-way handshake, idle chit-chat isn’t a strong point of geeks, since many don’t see the point (it’s a pragmatic thing). On the other hand, you could dive right into one of the topics of the day and that was also fine. Never mind that you hadn’t introduced yourself.

One thing I didn’t practice was the habit of a lot of people in the audience to spend the entire session on their laptops. Some were live-blogging the sessions, some were chatting on IRC about the speaker (there was apparently at least one instance of buzzword-bingo during a keynote), and who knows what others were doing. It wasn’t particularly distracting, so it’s not a complaint. Since I didn’t know anyone at the conference, I didn’t have any incentive to join in. Plus, my goal was to focus on the content of the talks, along with how it was being presented.

Let’s see, conference topics are the next thing to cover, but I’ll save that for next time.

O’Reilly’s Open Source Convention is now over for 2008, and I now have the rest of the day to relax, rest, and reflect before my flight tomorrow.

I ended up doing a lot of reflecting, so in the interest of not publishing a novel in one chapter, I broke this post up into several parts, and will post them more or less daily (if I remember) over the next few days.

Christine commented that I’ve effectively been off the grid this week, and that’s been mostly true. I’ve handled a few E-Mails, mostly customer-related, but otherwise I’ve been a lot less available than even the last time I was here.

That’s mildly odd, because I’ve also been a good bit less social this time around (last time I hung out a lot with the PostgreSQL folks; this time, other than an evening with the Mozilla QA people and a Birds-of-a-Feather session on open source in churches and missions, I barely spoke with anyone).

On the other hand, I was personally paying for this conference, whereas the last one was paid for by my former employer, so I had a lot more incentive to squeeze every penny of value out of the conference that I could. Not that I was slacking off the last time, by any stretch. Which is exactly why I’ve gotten a few “Hello?” E-Mails this week, I suppose.

I don’t think I learned as much this time as I did last time, on an absolute scale. That doesn’t really surprise me — I knew a lot less last time, and I’ve spent the past four years working on expanding on what I picked up from that conference. But the sessions at this conference did do a lot to fill in gaps and refine my knowledge, and would have been worth it for that reason alone.

One thing I learned last time was that it can be a better use of time to follow certain speakers around than just picking sessions off the chart based on topic. Quite simply, this is because some speakers can make any topic a worthwhile learning experience, while other speakers haven’t spent enough time learning about speaking to be able to effectively present even an interesting topic.

That was borne out this time as well. After the first day of the conference, I decided to follow Paul Fenwick of Perl Training Australia, to the extent that I watched him give the same talk twice. He was probably the best presenter at the conference this year, at least that I saw. His presentation style is fairly similar to Damian Conway’s (another person on my “follow” list).

A mildly related observation is that you know you picked a good (or at least an esoteric) session on Perl when Larry Wall is in the audience. For any CCC people reading this, that would be roughly akin to you giving a talk on the Four Spiritual Laws and having Bill Bright sitting in the first or second row, watching and listening attentively. So, Tim Bunce (of DBI fame) made it onto my “follow” list after that session, too. His presentations weren’t of the same caliber as the other two (few are), but they were very useful.

Speaking of usefulness, I skipped nearly all of the keynotes, except when Damian C. and Paul F. were presenting. Maybe it’s that open source is more of a pragmatic consideration for me than it is an ideological one, though I suspect that at least some of the keynotes were paid advertising. In any case, they didn’t seem all that interesting, and I heard at least a few others (who did attend) say the same thing. This had the added benefit that it let me sleep in.

Oh, and speaking of sleep, having a conference in Portland, OR was really nice once I got here. It meant that an 8:30am start time was actually an 11:30am start time as far as my physiology is concerned, which is a much better time for a morning session for geeks to start, in my opinion.

And that’s probably enough for one post. Observations on geek anthropology, conference topics, and presentation styles will be coming up subsequently.