What’s the hippie up to?

Father, it’s been a while since I last blogged…

Doesn’t mean nothing has been going on, not at all. Yesterday RPM 4.4.2.3 maintenance update was released with a fairly long list of bugfixes. The changelog has the details but various longstanding issues such as spaces in filenames, incorrect return codes from queries, “rpmbuild -bb –target <otherarch>” putting libraries to wrong directories on multilib systems etc have been fixed finally. This is the version that will be in Fedora 9, and it’s already in Rawhide.

Oh, I fully agree - maintenance updates to RPM 4.4.x aren’t particularly exciting. Maintenance releases of stable software are not supposed to be exciting! RPM isn’t the kind of software distros update lightly, it’s the very corner stone of RPM-based distributions: if it breaks, everything stops. So getting bugfixes to stable branch delivered in a predictable, no-surprises manner is of utmost importance. Exciting it is not.

Doesn’t mean there aren’t any exciting news though. RPM has just recently gotten more developer manpower behind it: Florian Festi and Jindrich Novy (both from Red Hat) are now working on RPM with me more or less full-time. This should give a nice boost to RPM development, cheers to Florian and Jindrich and my boss Denise who made this happen!

So, where’s the next major release lurking? Starting to be visible in the horizon, I would say. We’re still largely just clearing up ancient messes in the codebase - tedious, hard and largely thankless job of auditing and fixing the codebase for things like type- and const-correctness (rpm is/was full of code deliberately freeing data declared const for example), eliminating potential buffer overflows, splitting 1000+ line functions into smaller verifiable pieces, rewriting obscure code to be more readable, removing nasty surprises from the API and so on. Sexy new features - assuming one thinks there are such things in the world of RPM in the first place - are few and far between at the moment, we’ve been mostly removing “features” that don’t belong to RPM or are simply unsupportable. We want to make the ground solid before starting building anything new on it, even if it takes more time than one would like.

The time for actual new development work is closing in though, especially now with more folks actively working on RPM. Many things are in design phase, some already started like brand new Python bindings:

Rpm-python bindings are getting a thorough rewrite + redesign. The new bindings are being developed outside rpm proper and will be differently namespaced and parallel-installable to the current ones to allow for easy transition, the old in-rpm bindings will be maintained as long as necessary. While this does mean some extra (duplicated) work, it’s IMO the only way to get rid of various misdesigns, bad API’s and limitations of the current bindings without breaking the entire package build + management stack at once. The new bindings will be a mixture of Python and C with only low-level interface to librpm written in C, so if you want to hack on RPM stuff but only know Python, this is no problem anymore.

Other things that we’re looking at / going to look at include (but by no means limited to) multilib/arch dependencies, generic dependency filtering and custom dependency extractor mechanisms, redesigning the ancient transaction callback interface for extensibility, transaction performance and memory consumption issues, better API for header data handling … etc. The list is rather endless, and not all of it is going to happen now, or even this year. What I want to say here is that we’re not just sitting still on RPM 4.4.2.x, we’re working hard on delivering a solid new release, your patience is appreciated.

It’s only been little over a month since 4.4.2.1 was released, and I certainly didn’t expect to put out a new maintenance release out this soon but… the list of fixes has already grown long enough for it to make sense. RPM hasn’t exactly been known for quick bugfix delivery times, can’t hurt at least to try to change that by releasing more often - as often as it takes. So today I put out the first release candidate of RPM 4.4.2.2 maintenance release.

Ralf Corsépius has been doing a major cleaning up the cluttered and twisted make-system of rpm and also developing an automake-based test-system in the development tree. Meanwhile, I’ve been mostly trying to stay out of his way to minimize disruption and been hunting through bugzilla for existing patches which have been gathering dust there for ages; those fixes make up a large part of 4.4.2.2-rc1 contents.

As a result of these activities and getting RPM 4.4.2.1 delivered to FC6 and F7 as an update, the RPM bug-count for Fedora in has gone down considerably. It’s still way too high (over 90 open bugs) but that’s roughly 33% lower than it was in May. It might be nothing to write home about, at least progress is being made and folks getting their bugs fixed.

Oh and naturally the RC is soon to be found in Fedora Development repositories near you, it’s already packaged up and built and waiting just for repository push.

To clear up any false expectations: Rahul, contrary to what you wrote, RPM 4.4.2.1 doesn’t “fix the YUM CTRL-C issue”. If you read carefully what I said, that work has been done in rpm.org development branch, and will almost certainly be backported to the 4.4.x branch in the next maintenance release.

That said, the relevant patches are already included in Fedora Development RPM and thus will be in Fedora 8 starting from test1. But the RPM part is just an enabler, additionally YUM needs to make use of the new rpm-python methods before CTRL-C (and other normal exit signals) start working as expected. Initial patch to do this has just gone into YUM upstream but by the looks of things, F8-test1 wont yet have it as test1 has just been frozen.

Update: Seth says that thanks to a hiccup in the test spin he was able to slip the patch into test1 after all, excellent…

At a long last, RPM 4.4.2.1 is finally out after three release candidates and several weeks of testing in Fedora Development repository. It’s only a maintenance release, but rather significant in that it is the first RPM release from rpm.org since the “upstream reboot” in December 2006. For the glory details, see the 4.4.2.1 announcement on rpm-announce list.

Many moons have passed since 4.4.2 release, and thus the list of included fixes and cleanups is longer than one would perhaps otherwise expect from a minor maintenance release such as this. The list could’ve easily been much longer, but there’s a limit in how many changes one wants to include in what’s supposed to be a very stable and mature release. And besides, while the main focus will now be in the next major release, it’s quite likely that we’ll put out another minor maintenance release in the meantime to bring in some of the accumulated work from rpm.org development branch. Such as the support for getting YUM to honor CTRL-C again…

Oh and naturally the new version is soon to be found in Fedora Development repository, and eventually will be made available as an update to at least Fedora 7 and possibly Fedora Core 6 too.

Spent most of today fixing up things that cause gcc to whine about RPM source. Comparing rpm-4.4.2.1-rc3 vs current rpm.org head:

[pmatilai@localhost rpm]$ grep warning build.warnings.4421|wc -l
871
[pmatilai@localhost rpm]$ grep warning build.warnings|wc -l
362

Of the remaining 362 warnings, 252 come from Berkeley DB, so just over 100 more warnings to deal with still in actual rpm code. Of those, roughly a third come from debugedit.c alone… In any case, the build looks a whole lot less scary than it used to. Another big cleanup-round or two (these numbers are with -fno-strict-aliasing, removing that reopens another can of worms) and it might start to be possible to actually spot warnings from new code as it gets written.

Noticed with some amusement that “my” decision to revert the provides as implicit obsoletes behavior in rpm.org has been classified as “Panu’s Big Mistake” at rpm5.org: http://rpm5.org/cvs/chngview?cn=7632

Jeff has been insisting that the implicit obsoletion is intended and desirable feature but everybody else disagrees - if you look at various distributions shipping rpm >= 4.2, practically all of them have reverted that particular change. Except for Red Hat, and if you know a little bit about RPM history, it’s not hard to guess why. Yes, RH based distros have survived this four year period, but there have been endless complaints about the behavior. Some from surprised users, but mostly from packagers who find the implicit obsoletes very much unexpected and frustrating behavior as it prevents all sorts of otherwise useful packaging schemes. It hurts especially compatibility packages but there are other cases as well.

Sure, it was me who eventually applied the behavior reverting patch, but only following a lengthy discussion on the matter where the only concern people really had was the timing of the change, not the change itself. Various folks have been actually applausing the decision. The mistake here is firmly in the eye of the beholder…

Having seen DaveJ’s recent comment on RHEL-4 kernel having over thousand patches applied, I had this nasty feeling I’d seen a comment somewhere in the %prep parsing of rpmbuild stating “we can only handle 1024 patches!” And sure enough, there it is in parsePrep.c… yet all the 1076 patches are applied.

Turns out there’s a little known feature in the %patch “macro”: it can take several patch numbers at once, and that’s what the 1024 patch limit is about. You can use it like this: “%patch -p1 1 2 3 4″ instead of having four separate %patch lines.

Except that it’ll probably tell you “error: No patch number 0″ because %patch is treated as %patch0. So the above should be “%patch1 -p1 2 3 4″ to actually work. What a wonderfully broken feature, ugh.

For the last couple of months I’ve been pretty busy swatting bugs in rpm at work, and in the real life department, wasps inside and outside my apartment. The bastards have been trying to make a nest on my balcony, I disagree. The number of bugs fixed in rpm.org tree is by no means slim, but I suspect killed wasps outnumber the bug swats by largish margin still.

Anyhow, rpm 4.4.2.1 rc1 went out yesterday with a mile-long list of fixes and improvements. It was supposed to land in Fedora rawhide today but due to stupid mixups with the way rpm + popt packages are bundled together, that didn’t happen. Oh well, there’s always tomorrow…

I’ve been intending to start a blog on my open source activities for quite a while now, but never seemed to find anything to write about. Well, I switched jobs at beginning of May and am now working for Red Hat as RPM maintainer with Paul Nasrat. Pretty much all of my work happens in public repositories anyway so I can actually talk about my work in public now (a stark contrast to my previous employment) - if this isn’t a good time to start the bleeping blog finally, I don’t know what is.