The great filter of open source projects

This is how you make yourself vanish into nothing
— 24 frames, Something More Than Free, Jason Isbell and the 400 unit

So, with the recent layoffs at Mozilla — among other things — a bit of discussion on the sustainability of open source projects has been reignited. There was a wide range of takes: from “FOSS is dead” (no) to “we need to re-decentralize the internet” (yes). I could not quite help putting forth opinions on the matter myself and did so on a short twitter thread. Fundamentally though, the opinions expressed on this matter seem to almost talk past each other — and I think the reasons for this might be found in history of open source(1).

users are contributors

In the beginning, pretty much all open source projects had an almost complete overlap of users and contributors. Or at least potential contributors. E.g. the original GNU projects pretty much solved the problems of their own contributors and even most of those that where not contributors (yet) where non the less in the majority able to contribute to all parts of them. You can also see it in the original announcement of Linux as

“just a hobby, won’t be big and professional like gnu”

and while Linux grew way beyond that, there are still indications of this being true today when the main (and only) collaboration tool of the project, the LKML, was down for an extended time in 2018, because a mail server in a cabinet at someones home was not booting through after an power outage and the person to kick it was on vacation. The Linux kernel project was still self-hosting its infrastructure in 2018.

Another — later — project, that I am assuming to have been quite resilient and which I am assuming will continue to be quite resilient is gentoo linux: By requiring users to compile all software themselves, this distribution makes their users either give up on their installs or gets them at least halfway to be packagers (and for a distribution, packagers are contributors) themselves. Also, by not having to deal with binaries, gentoo reduces its infrastructure needs to a minimum. And even while there are some signs of downsizing at gentoo, I am hopeful that the flexibility mentioned above makes gentoo more sustainable and self-reliant than others for quite some time to come.

users are not all contributors anymore

In the 2000-2010 decade, especially in the second half, a lot of open source projects joined the choir were only a minority of users were contributors too. For many, the majority of users were not even possible contributors across the project. For a few, contribution was even implicitly limited to a closed circle. This gave rise to the perception that something like an “open source product” exists — especially by users. Here are some examples:

projectcomplementcontributors goal
Mozillawebprevent Microsoft from monopolizing the web, later generalized
OpenOffice.orgenterprise productivity
Androidweb/appsprevent Apple from monopolizing the web, app and smartphone market
Chromewebprotect a cloud, an advertising business and a search engine
Ubuntu desktopmanychanged over time, my guesses: initially, make launchpad what github is today, then complement the Ubuntu phone, then cloud offerings, but now something new maybe up

Of course, no such thing as an “open source product” exists and all of the above are variations of Strategy Letter V, which — being from 2002 — was already old by the time most of those were started:

Smart companies try to commoditize their products’ complements.

All of the above projects, commoditized their complements and this allowed users, who were not contributors to still benefit from the work of those who were as these contributors were interested in protecting the complement. With technology moving to the web and the cloud you can see this pattern repeating there in a few examples:

projectcomplementcontributors goal
reactfacebookensure all browser stay compatible with own use by making the framework widely used
istiocloud hostingraising the barrier to entry for microservice hosting (embrace and extend style)

the great filter of open source

So, what does this have to do with the layoffs at Mozilla and the current struggles at other open source projects? Well, the complements from strategy letter V that might motivate contributors to work on projects beyond their own needs exist at a given point in time, but … panta rhei, and this gives different outlooks for projects depending on how broad the complements are that it is serving:

Ultimately, open source projects provide a commodity. If their infrastructure needs are limited and their users also contributing they should be quite resilient (see e.g. gentoo). If they have many non-contributing users but have multiple complementing products, they will likely do well too.

However, having non-contributing users and only one complement might just be the Great Filter for open source projects: Once this complement is vanishing, so will the project — at least for non-contributing users. The remaining contributors — those that work without the need for a product, a complement, a business or users — will not feel that too much. But the non-contributing users will, as they wont be relevant at all anymore when the project downsizes itself to serving its contributing hobbyists alone.

conclusions and unrequested advice

So, at least for Mozilla and LibreOffice, I have some opinions and unrequested advice. In general, I see an urgent need for open source projects to establish a shared understanding among contributors what currently is a commodity — and thus is governed by the project and its institutions — and what are the complements of the commodity. For Mozilla and LibreOffice this might mean:

  • Felix von Leitner recently suggested at heise.de that the German government could make earmarked donations to Mozilla so protect digital sovereignty: “Die Bundesregierung könnte der Mozilla-Stiftung zweckgebundene Spenden zukommen lassen.”
    While I agree with both the goal and concerns over spending the money on a “cyber agency” might dilute results, earmarked donations are a also a very painful device to use: The administrative friction between a donor and the foundation will imply a huge overhead.
    Mozillas projects might be better off opening themselves to outside contributions and diversifying its contributions(2). And if there is indeed money in the German or European government to spend on strategic goals like digital sovereignty, tendering those to capable local providers with a crystal clear purpose will have much better outcomes. Both FSFE and OSBA might contribute starting points and help find suitable partners.
    All of this might seem very local, but likely it is not: Mozilla should look globally into diversifying the contributions to their projects, even if it might reduce their control as sole maintainer of them.
  • LibreOffice needs to decide where it really wants to provide a commodity and what its complements are: Both in the dimension of online collaboration vs. desktop and enterprise users vs. home users. The space claimed for commodity should be broad enough to motivate non-commercial contributors and to allow to grow into new complements when they appear, but overextending will leave too little room for complements, products and ultimately users. When the space allocated to be a commodity is overextended, the project will downsize to be a project solely serving its most active contributors(3) leaving aside non-contributing users.
    I see this responsibility at the board of the Document Foundation and there alone. It is elected for this as representatives of the community of contributors for exactly decisions like this.

Maybe not all of the dream is lost.

(1) Note that I am no expert on the history of open source nor do I know the internal workings and politics of all the open source projects, so there is unfortunately a lot of conjecture over the limited body of my own experience.

(2) This is where I might disagree with Michael Meeks’ take on that.

(3) The Document Foundation receives significant donations from individuals compared to other open source projects. Michael Meeks has some numbers on the value of developer contributions in kind to compare them to and put them in perspective.

Invitation to the LibreOffice Hackfest Hamburg 2018 on April 7th and 8th

… for a holiday of working class
She’s a runaway of the establishment incorporated
She won’t cooperate
— Last of the American Girls, 21 Guns, Green Day

Its once again time to meet again in Hamburg for a LibreOffice Hackfest! We will meet with contributors to LibreOffice – old and new – and also welcome those who have not yet contributed to the project, but are interested in doing so. This will also be the first Hackfest TNG.

We will have EasyHacks prepared for anyone wanting to do their first code contribution to the project – and we will have mentors around who may provide guidance to every interested newcomer. An Hackfest is the easiest way to get involved for a curious newcomers to the project: we certainly can offer an interesting C++ codebase!

Not interested in coding? As the local German LibreOffice community joins this Hackfest, non-coding topics like documentation, QA and marketing especially in the German context will be discussed too. This is an opportunity to meet those active in the German community and plan and coordinate how to push the project, the product and the brand of LibreOffice in the German context. Mixing and mingling of the German community with the international developer community is a bonus for both: otherwise this only happens at the frantic and busy meetings at FOSDEM and the LibreOffice conference, where usually time is a very restricted resource for everyone.

Note that as usual support for travel and accommodation for the Hackfest is available from The Document Foundation as per its travel policy, so please feel invited to make use of that offer, especially if you have always been curious about the LibreOffice project and you want to get involved, but are not quite sure how: a Hackfest is a great way to learn that.

So – where will we meet? Here:

6fwd

This is 6Fwd – a beautiful meeting room at freiheit.com technologies, who are generously hosting us for this event. I am very excited about us being guests at this location and would like to thank freiheit.com for their sponsorship!

Details about the LibreOffice Hackfest Hamburg 2018 are kept on the TDF wiki page and will be continuously updated. If you are interested in joining the event please start adding your name there (especially if you are interested in TDF sponsoring travel and accomodation for you).

(If you are still unsure about joining the event, you can have a look at the wiki pages of previous events to get an idea. But really: the best way to learn about the project is going to a Hackfest, so dont miss this opportunity!)

LibreOffice 5.2.3 as snap from Day One

Meist scheint manches auf den ersten Blick unmöglich.
Manches ist es auch, doch es wäre tödlich, das selbst zu glauben solange noch nichts feststeht
und die Party zu verlassen, bevor sie losgeht.

— Die Sterne, Stell die Verbindung her

Yesterday, two nice things happened: For one, LibreOffice 5.2.3 has been released and secondly Ubuntu Core 16 has been released. But beyond that, something in the middle between these two has happened: LibreOffice 5.2.3 has been released to the stable channel of the snap store at the same day. Now LibreOffice has been in the snap store for some time — and has also been on the stable channel since the Ubuntu 16.10 release. But this is the first time the LibreOffice snap is released in sync with The Document Foundation announcing the general availability of the final downloads. This was possible even though I was on vacation yesterday: LibreOffice snap packages are now being build on launchpad, which simplifies a lot, and launchpad can be asked to populate the edge channel of the store. This is making life very easy. Having smoketested the amd64 build from that channel before, to release LibreOffice 5.2.3 to the beta/candidate/stable channels too all I had to do was push three buttons on a web interface and it was available to all.

Building on launchpad, I also had the opportunity to create builds for armhf and i386 along with the usual amd64 builds with little extra effort. If you are adventurous you are encouraged to test these builds too: Be aware though that these so far aren’t even smoketested, I havent looked at them at all yet, so use them at your own risk.

All in all, this is great progress: LibreOffice 5.2.3 is available to users of Ubuntu 16.10 and Ubuntu 16.04 LTS as a snap on the day of the upstream release. And beyond that on all other distributions where snap is available — quite a few these days.

Update: ICYMI here is how to get the LibreOffice snap: http://www.libreoffice.org/download/snap/ — although strictly speaking you dont need the --channel=beta option anymore now. I will fix that soon.

Merging Communities

Come together, right now
— Beatles, Abbey Road, Come together

So September, 28th 2016 is the 6th birthday of LibreOffice and at the recent conference, we took a picture of those who were there on day zero:

brno32

As you might notice, I am not on that picture — on day zero I was working at Oracle, and were surprised by the news — like many others in this picture:

brno30

This is everyone at this years LibreOffice conference who used to work on the OpenOffice.org codebase at StarDivision, Sun or Oracle. A few people are in both pictures: Caolán McNamara and Thorsten Behrens were with LibreOffice from the start, but also worked at the OpenOffice.org team in Hamburg at some point in time. Of those working on OpenOffice.org still when LibreOffice started, I was the first to join LibreOffice — I had quit my job for that. It was an exciting time.

Looking back, both of these groups appear small — mostly because merging them, the LibreOffice community became so much more that the sum of its parts:

brno23

And of course, while a lot of people were at the conference, not everyone could
join, so there are more contributors from each of these groups than are in the
pictures. This years “state of the project” presentation showed again that the members of The Document Foundation are a truly worldwide community:

tdf-members

So, like the branches of the different descendants of OpenOffice.org, the
contributors and communities did come together in Brno to push
LibreOffice forward as one!

LibreOffice 5.2.0.2 available in the snap store

‘Cause the players gonna play, play, play, play, play
And the haters gonna hate, hate, hate, hate, hate
— Taylor Swift, 1989, Shake It Off

The latest release candidate of the upcoming LibreOffice 5.2.0 feature release is available for installation from the snap store. This makes it very easy to install this prerelease of LibreOffice for testing out new features (an incomplete glimpse on what to look forward for can be found on the LibreOffice 5.2 release notes page, which is still under construction, go on #libreoffice-qa if you want to help with testing).

To install this build of LibreOffice on any snap supported platform just open a terminal and run:

sudo snap install --channel=beta libreoffice

To start this version of LibreOffice, you run:

/snap/bin/libreoffice

The full path should only be needed, if you have another version of LibreOffice installed. If that is not the case a plain “libreoffice” should do.

Note that this version is still a prerelease and not for production use yet. That said, it is mostly a full-featured package including everything that would be packaged for end users of LibreOffice. While this package also includes a set of localizations to show that they work, their number has been restricted to English, French, German, Italian, Portuguese (Portugal/Brazil), Spanish for size considerations for now. This set is mostly the one Ubuntu provides on its installer images (removing those that might have issues as they need special fonts).

Another difference to prior downloads is that while LibreOffice still uses X11, now runs in confinement provided by snaps. Unlike previous releases on Ubuntu, this package defaults now to do so via the newer GTK3 backend: This has a lot of advantages, see details on Caolans Blog, but it is also a younger backend, that hasnt has that much time to be polished yet.

A third of a LibreOffice snap

Take your time, hurry up
The choice is yours, don’t be late
Take a rest as a friend
— Nirvana, Come As You Are

I have just updated the LibreOffice snap package. The size of the package available for download created some confusion. As LibreOffice 5.2 is still in beta, I built and packed it with full debug symbols to allow analysis of possible problems. Comparing this to the size of e.g. the default install from Ubuntu *.deb packages is misleading:

  • The Ubuntu default install misses LibreOffice Base and Java unless you explicitly install them
  • The Ubuntu default install misses debug symbols unless you install the package libreoffice-dbg too

As many people are just curious about running LibreOffice 5.2 without wanting to debug it right now, I replaced the snap package. The download and install instructions are still the same as noted here — but it is now 287MB instead of 1015MB (and it still contains Base, but no debug symbols).

The package file including full debug symbols — in case you are interested in that — has been renamed to libreoffice-debug.

(Note that if you downloaded the file while I moved files around, you might need to redo your download.)

LibreOffice 5.2.0 beta2 as a snap package

What’s been happening in your world?
What have you been up to?
— Arctic Monkeys, Snap out of it

So — here is what I have been up to:

LibreOffice 5.2.0 beta2 installed as a snap on Ubuntu 16.04
LibreOffice 5.2.0 beta2 installed as a snap on Ubuntu 16.04

The upcoming LibreOffice 5.2 packaged as a nice new snap package. This:

  • is pretty much a vanilla build of LibreOffice 5.2 beta2, using snapcraft, which is making packaging quite easy
  • contains all the applications: Writer, Calc, Impress, Draw, Math, Base
  • installs easily on the released current LTS version of Ubuntu: 16.04
  • allows you to test and play with the upcoming LibreOffice version to your hearts delight without having to switch to a development version of Ubuntu

So — how can you “test and play with the upcoming LibreOffice version to your hearts delight” with this on Ubuntu 16.04? Like this:

wget http://people.canonical.com/~bjoern/snappy/libreoffice_5.2.0.0.beta2_amd64.snap{,.sha512sum}
sha512sum -c libreoffice_5.2.0.0.beta2_amd64.snap.sha512sum && sudo snap install --devmode libreoffice_5.2.0.0.beta2_amd64.snap
/snap/bin/libreoffice

and there you have a version of LibreOffice 5.2 running — for example, you can prepare yourself for the upcoming LibreOffice Bug Hunting Session. And its even quite easy to remove again:

sudo snap remove libreoffice

This is one of the things that snap packages will make a lot easier: upgrading or  downgrading versions of an application, having multiple installed in parallel and much more. Watch out as there are more exciting news about this coming up!

Update: As this has been asked a few times: Yes, snap packages are available on Ubuntu. No, snap packages are not only available on Ubuntu. This text has more details.

Update 2: The original download included debug symbols and thus was quite big. The download now has 287MB. This post has all the details.

Zu Spät: Hackfest Hamburg

Warum hast Du mir das angetan?
Ich hab’s von einem Bekannten erfahren.

— Die Ärtze, Debil, Zu Spät

Its been more than two years since the last Hackfest in Hamburg! So we are indeed much too late (german: Zu Spät) with repeating this wonderful Event. Right a day after everyone updated his or her Desktop to Wily Werewolf we will meet for a weekend of happy hacking again in Hamburg!

Hamburg Hackfest 2013 - carelessly stolen from Eikes Retrospective
Hamburg Hackfest 2013 – carelessly stolen from Eikes Retrospective

So now, we will meet again. You are invited to drop by this weekend, we will celebrate a bit on Friday evening (ignoring the german culinary advise in the song linked above about “Currywurst and Pommes Fritz” — I imagine we prefer Club Mate and Pizza) and hack on LibreOffice on Saturday and Sunday. Curious new faces are more then welcome!

Blast from the Past: the Pimpl?

They sentenced me to twenty years of boredom
For trying to change the system from within
— Leonard Cohen, I’m your man, First we take Manhattan

Advance warning: This blog post talks about C++ coding style, and given the “expressiveness” (aka a severe infection with TimTowTdi) this is bound to contain significant amounts of bikeshedding, personal opinion/preference. As such, be invited to ignore all this as the ramblings of a raging lunatic.

Anyone who observed me spotting a Pimpl in code will know that I am not a fan of this idom. Its intend is to reduce build times by using a design pattern to move implementation details out of headers — a workaround for C++s misfeature of by default needing a recompile even for changing implementation details only without changing the public interface. Now I personally always thought a pure abstract base class to be a more “native” and less ugly way to tell this to the compiler. However, without real testing, such gut feelings are rarely good advisors in a complex language like C++.

So I did some testing on the real life performance of a pure abstract base class vs. a pimpl (each of course in a different compilation unit to prevent the compiler to optimize away what we want to measure) — and for reference, a class with functions that can be completely inlined. These are the three test implementations, inline:

-- header (hxx) --
class InlineClass final
{
	public:
		InlineClass(int nFirst, int nSecond)
			: m_nFirst(nFirst), m_nSecond(nSecond), m_nResult(0)
		{};
		void Add()
			{ m_nResult = m_nFirst + m_nSecond; };
		int GetResult() const
			{ return m_nResult; };
	private:
		const int m_nFirst;
		const int m_nSecond;
		int m_nResult;
};

Pimpl, as suggested by Effective Modern C++ when using C++11, but not C++14:

-- header (hxx) --
#include <memory>
class PimplClass final
{
	public:
		PimplClass(int nFirst, int nSecond);
		~PimplClass();
		void Add();
		int GetResult() const;
	private:
		struct Impl;
		std::unique_ptr<Impl> m_pImpl;
};
-- implementation (cxx) --
#include "pimpl.hxx"
struct PimplClass::Impl
{
	Impl(int nFirst, int nSecond)
		: m_nFirst(nFirst), m_nSecond(nSecond), m_nResult(0)
	{};
	const int m_nFirst;
	const int m_nSecond;
	int m_nResult;
};
PimplClass::PimplClass(int nFirst, int nSecond)
	: m_pImpl(std::unique_ptr<Impl>(new Impl(nFirst, nSecond)))
{}
PimplClass::~PimplClass()
	{}
void PimplClass::Add()
	{ m_pImpl->m_nResult = m_pImpl->m_nFirst + m_pImpl->m_nSecond; }
int PimplClass::GetResult() const
	{ return m_pImpl->m_nResult; }

Pure abstract base class:

-- header (hxx) --
#include <memory>
struct AbcClass
{
	static std::shared_ptr<AbcClass> Create(int nFirst, int nSecond);
	virtual ~AbcClass() {};
	virtual void Add() =0;
	virtual int GetResult() const =0;
};
-- implementation (cxx) --
#include "abc.hxx"
#include <memory>
struct AbcClassImpl final : public AbcClass
{
	AbcClassImpl(int nFirst, int nSecond)
		: m_nFirst(nFirst), m_nSecond(nSecond)
	{};
	virtual void Add() override
		{ m_nResult = m_nFirst + m_nSecond; };
	virtual int GetResult() const override
		{ return m_nResult; };
	const int m_nFirst;
	const int m_nSecond;
	int m_nResult;
};
std::shared_ptr<AbcClass> AbcClass::Create(int nFirst, int nSecond)
	{ return std::shared_ptr<AbcClass>(new AbcClassImpl(nFirst, nSecond)); }

Comparing these we find:

implementation lines added for GetResult() source entropy added source entropy for GetResult() runtime
inline 2 187 17 100%
Pimpl 3 316 26 168% (174%)
pure ABC 3 295 (273) 19 (16) 158%

So the abstract base class has less complex source code (entropy)1, needs less additional entropy to expand and is still faster in the end on common hardware (Intel i5-4200U) with common compiler optimization switches (-O2)2.

Additionally, in a non-trivial code base you might actually need to use virtual functions for your implementation anyway as you are deriving from or implementing an existing interface. In the Pimpl case, this means using two indirections (resolving the virtual function and then resolving the m_pImpl pointer in that function on top of that). In the abstract base class case thats not happening and in addition, it means that you can spare yourself the pure virtual declarations in the *.hxx (the virtual ... =0 ones), as those are already declared in the class derived from. In LibreOffice, this is true for any class implementing UNO interfaces. So the first numbers are actually biased against an abstract base class for real world code bases — the numbers in parathesis show the results when an interface is already defined elsewhere.

So unless the synthetic example used here is some kind of weird cornercase, this suggests abstract base classes being the better alternative over a Pimpl once the class goes beyond being a plain value type with completely inlineable accessor member functions.

Thanks for bearing with me on this rant about one of my personal pet peeves here!

1 entropy is measured as cat abc.[hc]xx|gzip|wc -c or cat pimpl.[hc]xx|sed -e 's/Pimpl/Abc/g'|gzip|wc -c.
2 Here is the code run for that comparision:

constexpr int repeats = 100000;

int pimplrun(long count)
//int abcrun(long count)
{
        std::vector< std::shared_ptr<PimplClass /* AbcClass */ > > vInstances;
        vInstances.reserve(count);
        while(--count)
                vInstances.emplace_back(std::make_shared<PimplClass>(4711, 4711));
                //vInstances.emplace_back(AbcClass::Create(4711, 4711));
        int result(0);
        count = vInstances.size();
        while(--count)
                for(auto pInstance : vInstances)
                {
                        pInstance->Add();
                        result += pInstance->GetResult();
                }
        return result;
}

Instances are stored in shared pointers as anything that a Pimpl is considered for would be “heavy” enough to be handled by reference instead of by value.

Update 1: Out of curiosity, I looked a bit deeper at this with callgrind. This is what I found for running the above (with 1000 repeats) and --cache-sim=yes:

I1 cache: 32768 B, 64 B, 8-way
D1 cache: 32768 B, 64 B, 8-way
LL cache: 3145728 B, 64 B, 12-way

event inline ABC Pimpl
Ir 23,356,163 38,652,092 38,620,878
Dr 5,066,041 14,109,098 12,107,992
Dw 3,060,033 5,094,790 5,099,991
I1ir 34 127 29
D1mr 499,952 253,006 999,013
D1mw 501,636 998,312 500,097
ILmr 28 126 24
DLmr 2 845 0
DLmw 0 1,285 250

I dont know exactly what to derive from that, but what is clear is that purely by instruction counts Ir this can not be explained. So you need --cache-sim=yes which gives the additional event counts. Actually Pimpl looks slightly better on most stats, so as it is slower in real life, the cache misses on the first level data cache D1mr might have quite an impact?

Update 2: This post made it to reddit, so I looked into some of the feedback from there. A common suggestion was to use for(auto& pInstance : vInstances) instead of for(auto pInstance : vInstances) in the benchmarking function. This had no significant impact on walltime measurements nor made it callgrind event counts show some clearer picture. I also played around with the order of linked objects to see if it has any impact (via cache locality etc.). While runtime measurements fluctuated quite a bit (even when using the same binary), the order was always the same: inlining quickest, then abstract base class and pimpl slowest.

Going mobile

When I’m drivin’ free, the world’s my home
When I’m mobile

— The Who, Who’s Next, Going Mobile

As you might have noticed, work has started to integrate LibreOffice with the document viewer of Ubuntu core apps. Here is a screenshot of how the current code renders documents on a mobile device:

Ubuntu core apps: LibreOffice and document viewer

Kudos for integrating this go entirely to Stefano Verzegnassi, all I did was providing a tiny piece of example code. It loads a document and saves a rendered version of the document to a PNG file. The relevant part of that piece of C++ code is small enough to fit in one picture shown here, including build instructions et al., showing how easy it is to use LibreOfficeKit from outside LibreOffice now:

 libreoffice2png source code

Thus the doc viewer was quickly integrated with LibreOffice in a basic way. This proof of concept isnt finished however: It just renders the all the document in one buffer. For small documents, this is reasonable, for bigger documents, tiled rendering — which LibreOfficeKit nicely supports from the API by allowing you to render any part of a document in a buffer — needs to be implemented on the clientside. The code for this can be found on launchpad, so if you are just curious how this works you are invited to have a look. If you are interested in helping out with moving this forward towards a nice all-around document viewer reading and rendering everything LibreOffice can, you are most welcome!

Update: A picture says more than a thousand words, but a video tells a whole story. Stefano created this awesome video, which you shouldnt miss: