Quick announcement: I'm going to be making two appearances over
the next week! First I'll be
two talks on September 30th at Red Hat's Chicago office for the
Chicago GNU/Linux User Group on Guix and Federation (both
mentioned in this post)! Second I'll be attending the
FSF 30th Birthday Party
in Boston on October 3rd. If you're able to make it to either,
do stop by and say hello... I'm expecting both to be a lot of fun!
Hello everyone! It's been a while since a comprehensive update of
what's happening in MediaGoblin land. Despite the quiet, there is a
lot to report, so let's get down to business and start reporting!
O'Reilly Award (again!)
Photo taken by Brandin Grams,
CC BY 4.0,
originally microblogged by
First of all, something fun: I was fortunate enough to receive the
O'Reilly Open Source Award! (Yes, I know about the terminology
mismatch, we're free software people, but it's still a great honor,
and I was presented the award under the description of my free
software advocacy and GNU MediaGoblin work.) The
other recipients of the award
is quite the incredible group of people, and I'm honored to be listed
among them. But here's what's really cool: you may remember that
MediaGoblin co-founder Deb Nicholson won the O'Reilly Award last year.
How's that as a vote of confidence in the things we're working on?
Anyway, if you want a more personal reflection, I
more on my personal blog!
So right, what about shipping software out the door? Well...
Since the crowdfunding campaign, we've gotten out two major releases,
0.7.0: Time Traveler's Delight
0.8.0: A Gallery of Fine Creatures.
I'm extremely proud of both of these releases! We have a lot more to
do though on the road to 1.0, and we've been directly been putting the
funds from the campaign to work to achieve that goal, so let's talk
Putting your money to good work: Jessica and Federation
You may recall that
we hired on the talented Jessica Tallon
to get federation working in MediaGoblin. Jessica recently
gave an update on the state of federation.
Jessica is doing great work, though as expected, converting
MediaGoblin to be a federated project has been no small task (knowing
what a big task it was, hope that we could hire Jessica on to do this
work was my #1 goal in the last campaign, in fact!). This decision
has turned out to be absolutely the right one. Some of the best parts
of the last two releases have been adopting the client to server Pump
API. Federation has been MediaGoblin's goal since day 0, and Jessica
is helping us to actually get there.
However (and now I'm going to do a pretty technical deep dive, so you
can skip this paragraph if that isn't your thing), the most
complicated aspect to making MediaGoblin into a federated project has
had to do with updating the database to handle things while preserving
data correctly for existing users. Why is this so complicated? A
number of years ago
we switched MediaGoblin over from using MongoDB to using either PostgreSQL or SQLite
and while I believe this was absolutely the right decision,
adding federation made the relational database system we had in place
substantially trickier. For the more database-technically inclined,
you can see that the
ActivityPump API /
require that any ActivityStream type object (in our case, that can be
media or comments or even users) be referenceable by any type of
activity. Furthermore, our existing comment system simply held that
comments referenced media entries, whereas now comments can reference
simply anything that is an ActivityStreams object. This means a
large portion of our relations in our relational database needed an
overhaul, and we needed a way to handle generic relations between
tables. (The solution used is not unlike the "generic foreign key"
implementation in Django.) There are more technical details on what
has been done, but Jessica has been neck deep in this for months, but
we believe we're finally on the home stretch, in which case Jessica
can finally knock out server to server federation.
(I've thought that a whole post on database structure lessons learned
may be a good blogpost of its own. One thing I'd note is that if
had been an option when our current database design was put together,
adopting that would have simplified things greatly, though it would
require being PostgreSQL-only. But moving to that now would require a
massive overhaul. If you're starting a new federated project from day 1,
maybe keep that in mind!)
So the summary of all the federation stuff is: it's complex, but we're
making good progress through Jessica's hard work. Expect more on this
soon, and huge strides in the next release!
Federation and the W3C Social Working Group
So something Jessica and I have both been involved in over the last
year is the
W3C Social Working Group
working towards official standards for federated web applications.
W3C Social Working Group
2015 face to face meeting attendees. Jessica's holding the laptop
on the left, and I'm right behind her.
Photo taken by Aaron Parecki,
posted to the W3C wiki.
The federation protocol that MediaGoblin has been working towards
until this point is primarily based on the
Pump API, but
this is really just a semi-formalization of the interface for the
pump.io API. In the Social Working Group we are
working towards defining a new standard,
ActivityPump, which is
based off of the
standard. We're very excited with where this standard is going and
feel it's a clean refinement over the Pump API we're already working
with, while still keeping many of those same conventions.
W3C Social Working Group
2015 face to face meeting attendees. I didn't make it to this
one, but Jessica did, and did a stellar job representing MediaGoblin
Photo taken by Aaron Parecki,
posted to the W3C wiki.
This has taken a lot of our time, but I believe it the results are
worth it. Jessica and I have been attending weekly calls related to
this standardization, and have thus far attended two face to face
meetings at well. (More accurately, Jessica attended the second
MediaGoblin-represented one without me, giving a kick-ass presentation
on how ActivityPump works to the group! Go Jessica!)
As for my own personal work advancing this, I'll go into this a bit
further on in this post!
Google Summer of Code result: Goblinoid!
Screenshot of Goblinoid, as it looks now!
Dylan Jeffers joined us for Google Summer of Code student this year
work on a pretty cool project: a MediaGoblin client for Android
or... really nearly anything... called
There's two really interesting features about Goblinoid: one, it's
written in Kivy, a GUI toolkit which emulates
the Android look and feel, but is actually can run nearly anywhere
Python can run... making it quite portable, yet ideal for mobile
Mockup of what Dylan would like Goblinoid to look like in
So Goblinoid works... it could use more user testing and packaging, if
you're interested in helping with that! But you can already upload
images on the go via Goblinoid, and we expect more to come.
Give it a go!
We've long been interested in having a client for MediaGoblin which
makes use of the Pump API implementation we've been working on.
Thank you Dylan for making that happen!
This has been a challenging year as in terms of supporting
MediaGoblin's infrastructure. Spammers attacked both our
bugtracker hard, at one point
leaving me to take several weeks to fight issues and to try to find
solutions. (Unfortunately, for the bugtracker, no great solutions
have been found, and we are on a request-an-account basis... not a
great situation to be in.)
Additionally, the primary Gitorious instance
went down, where MediaGoblin's code was hosted along with many, many
feature branches from contributors. The MediaGoblin community spent
a while debating what we were going to do. A move to GitHub was
tempting but is not an option; that's exactly the opposite of the type
of world we want to build. Not everyone in MediaGoblin's community is
comfortable with GitLab, and their primary instance is running a
proprietary version. There are some other communities hosting things
like Notabug who seem to be run by great
people, but we run the risk of running into these same problems all
over again... though so we did with most of these other solutions.
We could self-host, but there is very little time for extra server
maintenance burdens right now. So we
moved our git repository over to Savannah
and I'm glad to have hosting there by people I trust, though I do also
feel that contributors may expect more modern hosting facilities, but
we don't have time to run them ourselves.
The Gitorious shutdown came around the time of great exhaustion of
already dealing with issues related to the bugtracker and left me
feeling very burnt out. But it also lead to a great amount of
reflection... who am I to feel frustrated with? The Gitorious people
graciously hosted our software repositories for some time, and I was
not willing to run an instance of my own. Why not? Well, one problem
is that if you run your own instance of a free software web
application that isn't federated, you're working in Yet Another
Semi-Free Micro-Silo (TM). But let's face it, that's not the real
major problem... looking at the frustrations we've had with Trac, the
answer is obvious: running free network services is a huge
But wait... yes, you may be catching a whiff of irony here... if I'm
not willing to run a free software web application because I don't
want to take on the maintenance burden of someone else's software,
how can I ever expect MediaGoblin to gain adoption?
And here's where I come to a tough, but I think necessary, conclusion:
there's simply no way for MediaGoblin to succeed if the world of
deployment stays where it's at. Something must be done. But what?
Research into deployment and federation
Partly affected by the above, summer came around, and I had a talk
with MediaGoblin contributors in our monthly meetings. I wanted to
take a sabbatical... a sabbatical where I was on break from "direct"
MediaGoblin things, so I can advance things that affect MediaGoblin
greatly indirectly. (My spouse and I were also going through a big
move so this was a good time to do it, I figured.) I wanted to do
research into two things: deployment and federation.
Framing the deployment side of things, Deb Nicholson and I gave a talk
that kicked off "userops",
a term that I still feel accurately captures what we're trying to do
(and a talk which I still believe accurately summarizes the challenges
we're facing). Within this context I began exploring options of what
can be done to improve deployability.
The directions I've explored, and why I've come to the conclusions I
have, are a series of blogposts of their own (if you're interested, I
suggest subscribing to the
userops mailing list,
where I will be posting more as I go). The short of it is that I laid
out a set of requirements to achieve easy and maintainable
deployments, attempted to explore them with the most popular current
tools (Ansible, Puppet, Salt, Docker, etc), and found that it is not
possible to build both easy and maintainable systems on top of them
based on what I believe users need. This lead me eventually down the
path towards Guix, a package
manager (and with GuixSD, also a distro) and soon to be deployment
system which I am very confident has the potential to solve many of
the challenges which make deploying and maintaining systems too
exhausting a task for the average user. The software is nowhere near
being "easy" at present (I think it's very telling to say that it's
"the Emacs of package managers / distributions"... one can extrapolate
on that in many ways, most of which are correct, except for the
Emacs-haters kind... lay off, Emacs haters!) but I think has the
potential to become so. An easy to use web user interface has already
been demonstrated, and I believe the foundations are good for building
a complete and easy to use system for everyday users. But again, to
go into detail beyond what has already been said is something that
will be explored elsewhere.
There is a more pressing need for me to have explored deployment at
present as well... though I want to explore deployability for the sake
of other users, I also need to explore deployability for my own
sake... particularly because we promised that we will provide premium
hosting in the last campaign we ran! Yes, in case you have been
wondering about it, I have not forgotten about this promise. How to
fulfill this promise without being crushed by the maintenance burden
of hosting? I came to realize that there was every risk that I would
spend all my time supporting and maintaining servers that were running
MediaGoblin and I would not have the opportunity to any longer be a
steward of MediaGoblin itself... which could lead to failure. So
figuring out a better path forward on hosting has become a necessity.
I'll explore what this means further in the next section, but first, I
should say a word on federation and what my sabbatical lead to on this
As I have said, Jessica and I have been involved in the
W3C Social Working Group.
Part of the activities of the group have been the definition of
the ActivityStreams 2.0
and Pump API
I set out to explore this a bit more deeply under a repository
with the joking and interim name of
(the Social Working Group is using GitHub for its work, so I'm making
an exception there). Along with some other projects, this has
contributed to a deeper understanding of how federation should work,
which is something useful to take back to MediaGoblin. There are some
potentially useful things in that repository (including a partially
complete implementation of the
and it may turn into a full implementation of ActivityPump, though its
original and primary goal was for exploration, a purpose it has served
One major event relating to federation has just occurred over the last
week and bears note here: a number of Pump.IO
community members and myself are now working with Evan Prodromou (who
has near-certainly contributed more to the federated web space than
anyone else alive) to transition the project from being a project of
primarily stewardship under Evan to one of community stewardship and
governance. I posted a
summary of a recent meeting,
and (MediaGoblin contributor!) Laura Arjona put together a
In sum, Pump.IO could use your help if you're interested.
So anyway, it's been a busy last couple of months! But it's time to
return to MediaGoblin-ville, so...
Well, that's a whole lot of text above, so how about a bulleted list
next? I hear those are easier on the eyes.
I'm swooping back into MediaGoblin territory starting next month.
However, my initial focus will be on getting MediaGoblin to a
deployable point for myself, particularly forward-looking towards
making premium hosting feasible. Expect more soon!
Jessica is working on MediaGoblin federation. We expect the massive
database changes she has been working on to tidy up in the next few
weeks, and if all is well, we'll have server to server federation
basics in place by the end of the year.
Work towards 0.9.0 will proceed, unless 1.0 lands first!
Expect major infrastructure improvements in 0.9.0,
and federation to land in 1.0.
Those are all great things, and I think we are indeed on track towards
our goals, slowly but surely.
There is a more rough side to this: we've allocated nearly all of the
funds from the last campaign towards paying for Jessica's work on
MediaGoblin, and she's devoted enough to the project that she's been
working well, well below market rate. (And aside from some travel
reimbursements, I have not taken money personally from the last
campaign.) I'll post a financial transparency report soon, but the
short of it is: the MediaGoblin funds are running low. Even with
Jessica generously working for such a relatively low amount of money,
we won't be able to pay Jessica to work on MediaGoblin much longer,
given our present finances.
But we aren't giving up! The goals of MediaGoblin and of federating
the web are just too important, and so onward we press, into the
future! Though MediaGoblin is an ambitious project, I am confident we
can achieve the things we have set out to do, but we are constrained
by limited resources and time.
Appreciate what we are doing? Want to help us in our quest to bring
network freedoms to everyone? You can help!
The simplest way to help?
Though the official campaign
is over, you can still
donate to MediaGoblin through the FSF!
Jessica and I have been contracting so we can pay the bills (doing
this allowed us to "stretch out" the amount of time Jessica could
spend on the project... useful with all that W3C stuff in
progress!). Unfortunately, while the people we have been working
with are great, the contract we've been working on is coming to an
end. Are you looking for contractors or part time workers who are
capable of developing high quality free software and providing
community leadership? Jessica and I are both interested in working
in such cases... feel free to email me at:
cwebber AT dustycloud DOT org
We're committed to making a better, decentralized web. I hope this
piece cleared up where we're at and where we're going. I believe
we've got exciting times ahead... till next time, goblins!