See you at the Hackfest in Hamburg this weekend
With a rebel yell: “more, more, more”
More, more, more.
This weekend the LibreOffice community will meet again in Hamburg for the third Hackfest at this location:
here is how it looked last year:
- Hackers on the last Hamburg Hackfest
Like last year, this years Hackfest gets kicked off with a meet and greet at the Schachcafe on Friday 20:00 o’clock local time. Think of it like the “beer event” at FOSDEM, which helps everyone warm up for the event – except it will not be February and not freezing cold. Looking this picture, it likely wont be much like cold FOSDEM at all:
All details can be found on the Hackfest 2013 wiki page. Thanks to Lanedo for sponsoring this event and also big thanks to Attraktor.org for hosting us again!
One and Only
I am the one and only nobody I’d rather be
I am the one and only you can’t take that away from me
– Chesney Hawkes, The One and Only
Just a short note: I have missed the exact date in the release madness for Ubuntu 13.04 Raring, but a few days ago something important silently happened: All supported Ubuntu releases are now shipping with LibreOffice by default, as trusty old Ubuntu 10.04 LTS (Lucid Lynx) reached its end of support for the desktop. So we now have these supported releases:
- Ubuntu 12.04 LTS (Precise Pangolin) with LibreOffice 3.5
- Ubuntu 12.10 (Quantal Quetzal) with LibreOffice 3.6
- Ubuntu 13.04 (Raring Ringtail) with LibreOffice 4.0
- and upcoming: Ubuntu 13.10 (Saucy Salamander) with LibreOffice 4.1
Also the following releases (which are not supported anymore) have been done in addition:
- Ubuntu 11.04 (Natty Narwhal) with LibreOffice 3.3
- Ubuntu 11.10 (Oneiric Ocelot) with LibreOffice 3.4
Looking back in time at the angstridden, not-acting-but-reacting excitement of the early days and comparing it with the way we are really pushing the envelope now, we have really come a long way, improving with every step on the way. Well worth a celebration with one of the most cheesy 1990ies hits ever – Thanks to everyone, who was and is part of this!
LibreOffice prints on Tuesdays (only)!
Sitting on a cornflake, waiting for the van to come.
Corporation t-shirt, stupid bloody Tuesday.
– I am the walrus, The Beatles
Although the “OOo does not print on Tuesdays” OpenOffice.org-bug is long fixed, OpenOffice.org never indemnified Tuesday for its loss in reputation. This is unacceptable and Tuesdays rage at the event has even passed on to its successful successor: LibreOffice.
Now, Tuesday is a weekday and as such, money does not mean that much to it. After some consultation with Tuesday, it was concluded that the only way to indemnify Tuesday was to make the other weekdays suffer the same fate. Since the set of weekdays is luckily limited, this was easily considered technically feasible, even if at some minor inconvenience for the users of LibreOffice.
Toolbar popup of the new feature
To limit the impact for users, instead of silently failing to act when requested to print on a non-Tuesday, for the comfort of the users of LibreOffice a notification was added that explains the situation. Tuesday — after some heated discussion — gave license to this modification.
message on trying to print on non-Tuesdays
It is intended to ship this extension pre-bundled with all LibreOffice releases until Tuesdays rage is soothed. It is hoped this will happen quickly as both contributors and users of LibreOffice are known for their lack of sympathy for monopoly in any kind, shape or form — explicitly including the exclusive right of a weekday to print. For those of you excited to try out this thrilling new feature right now, the extension is available for download here:
It has to be noted that the extension is written purely in Python and is completely self-contained: it can either be treated as the oxt to be installed or as a zip file containing the source code: unpack it with the archive program of your choice, modify it to your hearts content and run the script called ‘build’ that you find in there. This will recreate a new (modified) extension.
autopkgtests for adults
“That’s not a knife — THAT’s a knife.”
I recently worked a bit to see this line showing up in my favorite editor:
ubtree0t-junit-subsequentcheck PASS
LibreOffice has multiple sets of testsuites and during the build of the package we run them all (although not yet on all platforms). However, LibreOffice depends on ~1/3 of main — so there are a lot of things that might break LibreOffice. A lot of things just break at build time and not at run time and thus prevent the such a broken package to enter the archive in the first place as we run the tests during the build already. Thats as: unless the breakage is caused by an update of a dependency of LibreOffice, therefore making the LibreOffice package in the archive FTBFS (or at least broken) in a sneaky way. Thats whats happened for the e.g. the libjpeg, boost, kdelibs examples above.
But lets keep those aside for now and concentrate on the runtime issues. Running the tests at build-time is a good early-warning already and prevents some serious breakage to enter the archive. On the other hand, these tests do not run against LibreOffice as we install it in the system from packages — it runs them against a installation set aside in the build tree. While I can not come up with an immediate example, where LibreOffice was broken when installed from the packages in a way that would have been detected by tests but missed when run against the in-tree installation, it would still be good to have the additional confidence that:
- LibreOffice passes the tests as installed on the system
- LibreOffice is not broken at runtime by some update of a dependency
In short: Its highly desirable to test that LibreOffice does still run and work as the ground below it keeps moving — this is even more important when Ubuntu is considering to move towards a more rolling way of releasing. And we have a means to do that: Autopkgtests.
So, what was needed to get this working for LibreOffice?
First, some parts of the testsuites are quite large and — as we run the tests during the build anyway — are already build during the build. Therefore it made sense to package these, which was done very early in the cycle (actually: during UDS).
Second, we would need to get LibreOffice to run the tests without trying to build the product. That originally wasnt as easy as it may seem. For one, LibreOffice build system was reasonably expecting that you need a product to test it and therefore would have dependencies on the product to be build. In addition, when I started considering this, we still had a lot of the old build system around — which was a pain to bend to your will. Luckily, these times are over. So, by now a patch changing some ~15 lines get us what we want.
Third, we need a config_host.mk (the output of ./configure), so that we can run the LibreOffice build. And for that, we unfortunately need the build dependencies (which are generated) of LibreOffice — otherwise we would not really test what we did build. But for a missing feature of autopkgtests, we can not reuse the existing dependencies, but have to do manual double bookkeeping there. Im not thrilled by the prospect of hunting false positives there. Some possible ways out would be:
- to package the config_host.mk file into the package containing the other testsuite helpers, but that would make that package architecture dependendant
- or to not really specify the dependencies at all and pragmatically and greedily request the restrictions needs-root and breaks-testbed and then — as we are root now — run this before starting the tests:
apt-get build-dep -y libreoffice
Finally, we should be able to run:
apt-get build-dep libreoffice apt-get install libreoffice-subsequentcheckbase apt-get source libreoffice cd libreoffice-* ./debian/tests/junit-subsequentcheck
and this should run the tests locally and headless — and indeed it does and the tests happily finish and report success. Great, lets quickly check if it also runs in the ‘official’ VM with:
run-adt-test
Nope, and this is why I choose the Crocodile Dundee quote below the title for this post: The VM fails before it even starts the tests — it does not even have enough discspace to copy in the LibreOffice source package. This needs to be fixed on the side of the image, there is nothing on the test side that could fix this. But to test if LibreOffice would finish if only the image could handle it, I began cannibalizing, removing one after another the directories of the icon-themes, translations and external sources from the package, each time getting a bit further: from failing to start to failing when installing the 501 additional packages and so on. With this hollowed out package, I could verify: yes, the autopkgtest would pass in the image, if only it had enough discspace.
Finally, once this is in the archive (or ppa) you will also be able to run:
apt-get build-dep libreoffice apt-get install libreoffice-subsequentcheckbase apt-get source libreoffice cd libreoffice-* libreoffice '--accept=pipe,name=blickenlights;urp'& ./debian/tests/junit-subsequentcheck 'connect:pipe,name=blinkenlights' 1
This will connect to the LibreOffice you started in the second-to-last step (which is not headless, but running in your session) and run the tests against it. The “1″ tells it not to use parallelization, but just run one suite at a time, as otherwise you have a very good chance to lock/hang your own session by compiz (or the dash or other components) being mightly confused by all the windows flashing up and closing in fast progression. With “1″ you might still get some test failures (mostly from the a11y integration) — but at least your session will survive:
ZO RELAXEN UND WATSCHEN DER BLINKENLICHTEN.
Addendum:
Preparing an adt-image with:
./bin/prepare-testbed -r raring amd64 -S12GB
seems to solve the issue. The “df -h” at the start of the test reports some 3GB of free space (with 2.6GB being needed still to create a rw-copy of the source tree after that point). So 12GB is likely the size the images on Jenkins roughly currently need (plus maybe another 1GB of wiggle room).
LibreOffice Debian/Ubuntu Team growing
“Ich danke der Academy für das Erkennen von Talent.”
-- Ich danke der Akademie, Kettcar
So, I just updated the LibreOffice version for Ubuntu Raring in the LibreOffice PPA to the 4.0.1~rc2 version, that was announced as 4.0.1 final yesterday. As we dropped the old binfilter file import upstream in LibreOffice 4, LibreOffice fits again in the ppa with all localizations — previously the PPA versions were restricted to a subset of languages, as the buildds for the PPAs might run out of disc space otherwise. Oh, and a boring sidenote just for completeness: LibreOffice on Ubuntu LTS has been getting a stable release update to 3.5.7 a while ago. And now that 4.0.1 rc2 is declared final, I expect it to be sponsored soon to Ubuntu Raring proper.
I would like to take this opportunity to thank everyone who made this possible:
- Rene Engelhard for his work on the Debian LibreOffice packaging, making all of this possible
- Boaz Dodin and Rico Tzschichholz for their tireless work and initiative on backporting current releases
- Christopher M. Penalver for churning through LibreOffice bugs in launchpad and on freedesktop, syncing and triaging them
- Benjamin Drung for fixes, tweaks and contributions to the LibreOffice packaging both at Debian and Ubuntu
Will there be backports of 4.0.1 to older series? Well, an indiscreet look in Ricos PPA, suggests that it wont take long for them to end up in the LibreOffice ppa. Steadily, the LibreOffice team is growing — its exciting to see so many volunteers contribute to this!
One
“One Ring to rule them all”
– J.R.R. Tolkien, The Lord of the Rings
LibreOffice is now build by one instance of make that is aware of the whole dependency tree. According to my master development build (that is: a build without localization, help, extensions) yesterday, this instance of make now knows about
126.501 targets from 1.717 makefiles
and has a complete view of how they relate to each other. The memory usage of make is at 207 MiB, only slightly overshooting the initial estimations done in the early days of gbuild of 170-190MiB (counting in that the codebase changed a lot in two years, the estimate is actually really good). Given that recursive make is considered harmful and that LibreOffice — one of the biggest open source projects, with huge dependencies and doing releases on three platforms (Windows, OS X and Unix — a lot more if you separate the different Unix flavours), can do this — there is little excuse left for other projects to not follow suit.
On my machine, checking if anything needs to be rebuild in LibreOffice now takes ~28.7sec (or 37.2sec when also running the default sanity checks along that). That might sound a lot, but consider the scale! And it is a long way from the old OpenOffice.org build system that we came from: Just from my memory, it took about 5 Minutes to do that on the old build system. On Windows it took almost 30 Minutes to find out that there is nothing to do. One of my earliest talks (Slide 29) on the topic of gbuild compared the performance of partial build, if you find these numbers hard to believe. Oh, and of course you still can check for updating only a subset of LibreOffice (a “module” – e.g. Writer) and that takes only 2-3 seconds even for the biggest ones.
How gbuild spends the 37 seconds to ensure that nothing need to be rebuild: orange = reading/parsing the makefiles (singlethreaded), grey = stat’ing and checking the filesystem, blue = running sanity tests (multithreaded)
Does this difference in performance matter? As Linus argued to eloquently in his google tech talk on git: Yes, it does. Because it enables different ways to work, that just were not possible before. One such example is that we can have incremental build tinderboxes like the Linux-Fedora-x86_64_22-Incremental one, which comes with a turnaround of some 3-5 minutes most of the time by now and quickly reports if something was broken.
There are other things improved with the new build system too. For example, in the old build system, if you wanted to add a library, you had to touch a lot of places (at minimum: makefile.mk for building it, prj/d.lst for copying it, solenv/inc/libs.mk for others to be able to link to it, scp2 to add it to the installation and likely some other things I have forgotten), while now you have to only modify two places: one to describe what to build and one to describe where it ends up in the install. So while the old build system was like a game of jenga, we can now move more confidently and quickly.
Touching the old build system was like a game of jenga. Except that it wasnt fun. (Photo: Copyright CC BY-NC-SA 2.0 Jose Hernandez)
Then there is scalability: The old build system did not scale well beyond 4-8 jobs as it had no global notion of how make jobs where running. As we see CPU architectures become more important that have slower, but cheaper cores, this is getting increasingly relevant. Do you have a 1000 core distcc cluster you want to testdrive? LibreOffice might be the project you want to try.
Finally, the migration to gbuild is a proof of how amazing the community is that is growing around the project: While I set up the initial infrastructure for gbuild, the hard work of migrating over 200 modules (each the size of your average open source project) to it without breaking on one of three platforms or disrupting the ongoing development on features and bugfixes was mostly done by a crowd of volunteers. Looking back, I doubt the migration to gbuild would have been completed in reasonable time in an environment less inviting to volunteers and contributors — it was the distribution of the work that made this possible. So the credit for that we now can profit from the benefits of gbuild really goes to these guys. Big kudos for everyone working on this, you created something amazing!
Addendum: This post has been featured on lwn and led to a spirited discussion there.
Notes:
For estimating the number of targets, I used:
make -f Makefile -np all slowcheck|grep 'File.*update'|wc -l
For the memory usage:
pmap -d $(ps -a|grep make|cut -f1 -d\ )|egrep -o writeable/private:.[0-9]+K|cut -f 2 -d\
Brothers in Arms
But it’s written in the starlight
And every line on your palm
We’re fools to make war
On our brothers in arms
– Dire Straits, Brothers in Arms
As the LibreOffice project grows, the distance between teams and subprojects also grows. This makes it increasingly important to keep our efforts united and not tolerate behaviour that makes members feel attacked,
even if the originators believe they are simply being direct or mischievous. We’re a diverse, multi-cultural and multi-lingual community, and it’s easy to unintentionally offend through choices of expression that differ from what’s anticipated.
We’re a single community, and creating the concept of others – and the promotion of a “them vs. us” mentality — in our team is dangerous and will hurt us in the long run. Some basic guidelines to this regard are set out in the statutes of the Document Foundation. If something more explicit is needed the Ubuntu code of conduct gives a good overview of what to take care of in the project.
Unfortunately, there recently have been repeated incidents on bugzilla which violate this principle. Joel has thus been given permissions to suspend users on bugzilla temporarily if it proves necessary. We hope that friendly corrections to tone and messaging, then warnings and finally the threat of such suspension will be enough, and there will be no need for action to be taken.
