Quick announcement: I'm going to be making two appearances over the next week! First I'll be giving two talks on September 30th at Red Hat's Chicago office for the Chicago GNU/Linux User Group on Guix and Federation (both mentioned in this post)! Second I'll be attending the FSF 30th Birthday Party in Boston on October 3rd. If you're able to make it to either, do stop by and say hello... I'm expecting both to be a lot of fun!
Hello everyone! It's been a while since a comprehensive update of what's happening in MediaGoblin land. Despite the quiet, there is a lot to report, so let's get down to business and start reporting!
O'Reilly Award (again!)
Photo taken by Brandin Grams,
CC BY 4.0,
originally microblogged by
Karen Sandler
First of all, something fun: I was fortunate enough to receive the O'Reilly Open Source Award! (Yes, I know about the terminology mismatch, we're free software people, but it's still a great honor, and I was presented the award under the description of my free software advocacy and GNU MediaGoblin work.) The other recipients of the award is quite the incredible group of people, and I'm honored to be listed among them. But here's what's really cool: you may remember that MediaGoblin co-founder Deb Nicholson won the O'Reilly Award last year. How's that as a vote of confidence in the things we're working on?
Anyway, if you want a more personal reflection, I wrote more on my personal blog!
Releases
So right, what about shipping software out the door? Well...
Since the crowdfunding campaign, we've gotten out two major releases, 0.7.0: Time Traveler's Delight and 0.8.0: A Gallery of Fine Creatures. I'm extremely proud of both of these releases! We have a lot more to do though on the road to 1.0, and we've been directly been putting the funds from the campaign to work to achieve that goal, so let's talk about that.
Putting your money to good work: Jessica and Federation
You may recall that we hired on the talented Jessica Tallon to get federation working in MediaGoblin. Jessica recently gave an update on the state of federation. Jessica is doing great work, though as expected, converting MediaGoblin to be a federated project has been no small task (knowing what a big task it was, hope that we could hire Jessica on to do this work was my #1 goal in the last campaign, in fact!). This decision has turned out to be absolutely the right one. Some of the best parts of the last two releases have been adopting the client to server Pump API. Federation has been MediaGoblin's goal since day 0, and Jessica is helping us to actually get there.
However (and now I'm going to do a pretty technical deep dive, so you can skip this paragraph if that isn't your thing), the most complicated aspect to making MediaGoblin into a federated project has had to do with updating the database to handle things while preserving data correctly for existing users. Why is this so complicated? A number of years ago we switched MediaGoblin over from using MongoDB to using either PostgreSQL or SQLite and while I believe this was absolutely the right decision, adding federation made the relational database system we had in place substantially trickier. For the more database-technically inclined, you can see that the ActivityPump API / Pump API require that any ActivityStream type object (in our case, that can be media or comments or even users) be referenceable by any type of activity. Furthermore, our existing comment system simply held that comments referenced media entries, whereas now comments can reference simply anything that is an ActivityStreams object. This means a large portion of our relations in our relational database needed an overhaul, and we needed a way to handle generic relations between tables. (The solution used is not unlike the "generic foreign key" implementation in Django.) There are more technical details on what has been done, but Jessica has been neck deep in this for months, but we believe we're finally on the home stretch, in which case Jessica can finally knock out server to server federation.
(I've thought that a whole post on database structure lessons learned may be a good blogpost of its own. One thing I'd note is that if jsonb had been an option when our current database design was put together, adopting that would have simplified things greatly, though it would require being PostgreSQL-only. But moving to that now would require a massive overhaul. If you're starting a new federated project from day 1, maybe keep that in mind!)
So the summary of all the federation stuff is: it's complex, but we're making good progress through Jessica's hard work. Expect more on this soon, and huge strides in the next release!
Federation and the W3C Social Working Group
So something Jessica and I have both been involved in over the last year is the W3C Social Working Group working towards official standards for federated web applications.
W3C Social Working Group
March
2015 face to face meeting attendees. Jessica's holding the laptop
on the left, and I'm right behind her.
Photo taken by Aaron Parecki,
CC0 1.0,
originally
posted to the W3C wiki.
The federation protocol that MediaGoblin has been working towards until this point is primarily based on the Pump API, but this is really just a semi-formalization of the interface for the pump.io API. In the Social Working Group we are working towards defining a new standard, ActivityPump, which is based off of the ActivityStreams 2.0 standard. We're very excited with where this standard is going and feel it's a clean refinement over the Pump API we're already working with, while still keeping many of those same conventions.
W3C Social Working Group
May
2015 face to face meeting attendees. I didn't make it to this
one, but Jessica did, and did a stellar job representing MediaGoblin
and ActivityPump!
Photo taken by Aaron Parecki,
CC0 1.0,
originally
posted to the W3C wiki.
This has taken a lot of our time, but I believe it the results are worth it. Jessica and I have been attending weekly calls related to this standardization, and have thus far attended two face to face meetings at well. (More accurately, Jessica attended the second MediaGoblin-represented one without me, giving a kick-ass presentation on how ActivityPump works to the group! Go Jessica!)
As for my own personal work advancing this, I'll go into this a bit further on in this post!
Google Summer of Code result: Goblinoid!
Screenshot of Goblinoid, as it looks now!
Dylan Jeffers joined us for Google Summer of Code student this year work on a pretty cool project: a MediaGoblin client for Android or... really nearly anything... called Goblinoid! There's two really interesting features about Goblinoid: one, it's written in Kivy, a GUI toolkit which emulates the Android look and feel, but is actually can run nearly anywhere Python can run... making it quite portable, yet ideal for mobile computers!
Mockup of what Dylan would like Goblinoid to look like in
the future!
So Goblinoid works... it could use more user testing and packaging, if you're interested in helping with that! But you can already upload images on the go via Goblinoid, and we expect more to come. Give it a go!
We've long been interested in having a client for MediaGoblin which makes use of the Pump API implementation we've been working on. Thank you Dylan for making that happen!
Infrastructure challenges
This has been a challenging year as in terms of supporting MediaGoblin's infrastructure. Spammers attacked both our wiki and bugtracker hard, at one point leaving me to take several weeks to fight issues and to try to find solutions. (Unfortunately, for the bugtracker, no great solutions have been found, and we are on a request-an-account basis... not a great situation to be in.)
Additionally, the primary Gitorious instance went down, where MediaGoblin's code was hosted along with many, many feature branches from contributors. The MediaGoblin community spent a while debating what we were going to do. A move to GitHub was tempting but is not an option; that's exactly the opposite of the type of world we want to build. Not everyone in MediaGoblin's community is comfortable with GitLab, and their primary instance is running a proprietary version. There are some other communities hosting things like Notabug who seem to be run by great people, but we run the risk of running into these same problems all over again... though so we did with most of these other solutions. We could self-host, but there is very little time for extra server maintenance burdens right now. So we moved our git repository over to Savannah and I'm glad to have hosting there by people I trust, though I do also feel that contributors may expect more modern hosting facilities, but we don't have time to run them ourselves.
The Gitorious shutdown came around the time of great exhaustion of already dealing with issues related to the bugtracker and left me feeling very burnt out. But it also lead to a great amount of reflection... who am I to feel frustrated with? The Gitorious people graciously hosted our software repositories for some time, and I was not willing to run an instance of my own. Why not? Well, one problem is that if you run your own instance of a free software web application that isn't federated, you're working in Yet Another Semi-Free Micro-Silo (TM). But let's face it, that's not the real major problem... looking at the frustrations we've had with Trac, the answer is obvious: running free network services is a huge maintenance burden.
But wait... yes, you may be catching a whiff of irony here... if I'm not willing to run a free software web application because I don't want to take on the maintenance burden of someone else's software, how can I ever expect MediaGoblin to gain adoption?
And here's where I come to a tough, but I think necessary, conclusion: there's simply no way for MediaGoblin to succeed if the world of deployment stays where it's at. Something must be done. But what?
Research into deployment and federation
Partly affected by the above, summer came around, and I had a talk with MediaGoblin contributors in our monthly meetings. I wanted to take a sabbatical... a sabbatical where I was on break from "direct" MediaGoblin things, so I can advance things that affect MediaGoblin greatly indirectly. (My spouse and I were also going through a big move so this was a good time to do it, I figured.) I wanted to do research into two things: deployment and federation.
Framing the deployment side of things, Deb Nicholson and I gave a talk that kicked off "userops", a term that I still feel accurately captures what we're trying to do (and a talk which I still believe accurately summarizes the challenges we're facing). Within this context I began exploring options of what can be done to improve deployability.
The directions I've explored, and why I've come to the conclusions I have, are a series of blogposts of their own (if you're interested, I suggest subscribing to the userops mailing list, where I will be posting more as I go). The short of it is that I laid out a set of requirements to achieve easy and maintainable deployments, attempted to explore them with the most popular current tools (Ansible, Puppet, Salt, Docker, etc), and found that it is not possible to build both easy and maintainable systems on top of them based on what I believe users need. This lead me eventually down the path towards Guix, a package manager (and with GuixSD, also a distro) and soon to be deployment system which I am very confident has the potential to solve many of the challenges which make deploying and maintaining systems too exhausting a task for the average user. The software is nowhere near being "easy" at present (I think it's very telling to say that it's "the Emacs of package managers / distributions"... one can extrapolate on that in many ways, most of which are correct, except for the Emacs-haters kind... lay off, Emacs haters!) but I think has the potential to become so. An easy to use web user interface has already been demonstrated, and I believe the foundations are good for building a complete and easy to use system for everyday users. But again, to go into detail beyond what has already been said is something that will be explored elsewhere.
There is a more pressing need for me to have explored deployment at present as well... though I want to explore deployability for the sake of other users, I also need to explore deployability for my own sake... particularly because we promised that we will provide premium hosting in the last campaign we ran! Yes, in case you have been wondering about it, I have not forgotten about this promise. How to fulfill this promise without being crushed by the maintenance burden of hosting? I came to realize that there was every risk that I would spend all my time supporting and maintaining servers that were running MediaGoblin and I would not have the opportunity to any longer be a steward of MediaGoblin itself... which could lead to failure. So figuring out a better path forward on hosting has become a necessity. I'll explore what this means further in the next section, but first, I should say a word on federation and what my sabbatical lead to on this front.
As I have said, Jessica and I have been involved in the W3C Social Working Group. Part of the activities of the group have been the definition of the ActivityStreams 2.0 and Pump API standards.
I set out to explore this a bit more deeply under a repository with the joking and interim name of activitystuff (the Social Working Group is using GitHub for its work, so I'm making an exception there). Along with some other projects, this has contributed to a deeper understanding of how federation should work, which is something useful to take back to MediaGoblin. There are some potentially useful things in that repository (including a partially complete implementation of the JSON-LD API), and it may turn into a full implementation of ActivityPump, though its original and primary goal was for exploration, a purpose it has served well.
One major event relating to federation has just occurred over the last week and bears note here: a number of Pump.IO community members and myself are now working with Evan Prodromou (who has near-certainly contributed more to the federated web space than anyone else alive) to transition the project from being a project of primarily stewardship under Evan to one of community stewardship and governance. I posted a summary of a recent meeting, and (MediaGoblin contributor!) Laura Arjona put together a community document. In sum, Pump.IO could use your help if you're interested.
So anyway, it's been a busy last couple of months! But it's time to return to MediaGoblin-ville, so...
What's next?
Well, that's a whole lot of text above, so how about a bulleted list next? I hear those are easier on the eyes.
- I'm swooping back into MediaGoblin territory starting next month. However, my initial focus will be on getting MediaGoblin to a deployable point for myself, particularly forward-looking towards making premium hosting feasible. Expect more soon!
- Jessica is working on MediaGoblin federation. We expect the massive database changes she has been working on to tidy up in the next few weeks, and if all is well, we'll have server to server federation basics in place by the end of the year.
- Work towards 0.9.0 will proceed, unless 1.0 lands first! Expect major infrastructure improvements in 0.9.0, and federation to land in 1.0.
Those are all great things, and I think we are indeed on track towards our goals, slowly but surely.
There is a more rough side to this: we've allocated nearly all of the funds from the last campaign towards paying for Jessica's work on MediaGoblin, and she's devoted enough to the project that she's been working well, well below market rate. (And aside from some travel reimbursements, I have not taken money personally from the last campaign.) I'll post a financial transparency report soon, but the short of it is: the MediaGoblin funds are running low. Even with Jessica generously working for such a relatively low amount of money, we won't be able to pay Jessica to work on MediaGoblin much longer, given our present finances.
But we aren't giving up! The goals of MediaGoblin and of federating the web are just too important, and so onward we press, into the future! Though MediaGoblin is an ambitious project, I am confident we can achieve the things we have set out to do, but we are constrained by limited resources and time.
Appreciate what we are doing? Want to help us in our quest to bring network freedoms to everyone? You can help!
Donate...
- The simplest way to help? Donate! Though the official campaign is over, you can still donate to MediaGoblin through the FSF!
- Jessica and I have been contracting so we can pay the bills (doing this allowed us to "stretch out" the amount of time Jessica could spend on the project... useful with all that W3C stuff in progress!). Unfortunately, while the people we have been working with are great, the contract we've been working on is coming to an end. Are you looking for contractors or part time workers who are capable of developing high quality free software and providing community leadership? Jessica and I are both interested in working in such cases... feel free to email me at: cwebber AT dustycloud DOT org
We're committed to making a better, decentralized web. I hope this piece cleared up where we're at and where we're going. I believe we've got exciting times ahead... till next time, goblins!