State of the Goblin: Stripe Open Source Retreat, and more!

Hello, all!

It's been a few months since my last major update so I wanted to fill in what's going on. As usual, a lot has been happening, and it's been hard to cover it all as we go. There's some particularly huge news in this update, including something about funding something oh hey this should help us get MediaGoblin 1.0 out the door, plus something about the standards work we're doing, something something. So let's dive in and resolve all those somethings, right?

Support organizations that support freedom!

Okay, wait, a brief intermission! We'll get to the cool MediaGoblin related news in a moment, but we've got something very important to cover first. Two organizations I really care about are running funding campaigns. Okay, well, it's that time of year, and a lot of organizations are running funding campaigns, but these two especially could use your kind contributions!

Support Conservancy!


Copyheart by Christopher Allan Webber, CC BY 4.0 or GPLv3 or later, your option.. Source here.
Become a Conservancy supporter!

The first is Software Freedom Conservancy. They're close friends of ours, and do great work. They're also pushing hard to try to build up a supporter program, and they could really use your help. Conservancy does a lot of work around many things, from running Outreachy (hey, we were lucky to get Jessica Tallon working on our stuff through that!) to enforcing the GPL to hosting a whole lot of useful free software projects. Become a supporter today!

Support the FSF!

Next up is the Free Software Foundation. Hey you probably know these folks right? The FSF is the steward of GNU, and MediaGoblin is a GNU project, so success for the FSF is success for MediaGoblin. They're running their annual fundraising campaign and they could really use your donation. The FSF has the long-standing history of being the anchor of the free software world, and they continue to do great work year after year. Help the FSF continue far into the future... get your donation in today!

Support Guix!


GuixSD logo by Luis Felipe López Acevedo. Permission is granted to copy, distribute and/or modify this work under CC BY 4.0.
Support Guix!

Did I say two organizations? What's with three sections? In fact, our friends at the FSF are also teaming up with our friends in the GNU Guix project to run a special campaign to raise funds for new servers. I'm highlighting this for two reasons:

1. I believe Guix is very important to the long run of "deployability" in free software network services (which you may know I believe to be an extremely important issue...)
2. Guix is using a similar funding model of going through the FSF as fiscal sponsor. This is the same route we took for the MediaGoblin campaigns and I think is a great way for free software projects, and particularly GNU projects, to go. I'd love to see further examples of success for it in Guix.

So, go donate! :)

Okay, whew... sorry about that distraction, but these things are really important! But I know, I know, you came here for the MediaGoblin news. Well, let's get to that!

MediaGoblin selected for the Stripe Open Source Retreat

So this is some big news! MediaGoblin has been selected to be a participant in Stripe's Open Source Retreat 2016!

What does this mean? It means that I'll be moving to San Francisco from mid-January to mid-April 2016, working from Stripe's office, and Stripe is going to pay me to focus totally on getting MediaGoblin 1.0 out the door and advancing our federation work. This is a huge opportunity for us; getting such ~unrestricted funding is, as anyone who has ever done fundraising knows, enormously difficult. We should be able to use this to bring MediaGoblin to the next level.

When I filled out the application for this I was interested but skeptical of Stripe's claim that this would be "no strings attached" funding. I'm happy to say this seems to be true: they've paid us to do the work, and Stripe is making no claim to copyright or asking us to change any of our existing policies. Contributing upstream to MediaGoblin happens as usual for the time I'm there, which is great.

So what do we hope to get out of it? Well, my goal is by the end of the retreat, we'll have MediaGoblin 1.0 out the door with the basics of server to server federation in place. I've also talked with the Stripe team about using that time to advance work on our federation standards work, and if there's time, some deployment work too. But MediaGoblin 1.0 comes first!

(In the meanwhile I just booked a bunk bed for $1000 USD per month, which it turns out is cheap for San Francisco housing. Egads! How do people afford to live there? Luckily it's well covered by the retreat's stipend! Are you in the area during that time? Maybe we should meet up!)

I'm very excited about this opportunity. Thanks again to Stripe for supporting our community. I promise that the grant will go to good use, and we'll have exciting things to report!

W3C updates


W3C Social Working Group December 2015 face to face meeting attendees. I'm hiding in the back.
Photo taken by Aaron Parecki, CC0 1.0, originally posted to the W3C wiki.

Our work to standardize federation technology within the Social Working Group continues. Just a few weeks ago another face to face meeting was held at Mozilla's San Francisco offices (thus giving me an excuse to test-run the bunk bed I'll be sleeping in for several months for the Stripe retreat). The meeting was extremely productive... in fact I would say it was the most productive time the Social Working Group has ever had. You can read the minutes for Day 1 and Day 2 if you like that sort of thing. But here are some highlights:

• Most of the work towards making ActivityStreams 2 a Candidate Recommendation document was put in place, and in a couple of weeks this I expect it will achieve that goal. Candidate Recommendation is a big step in a standards process, and AS 2.0 is the heart of ActivityPump, so this is huge for us.
• ActivityPump, the standard we are pushing for server to server federation and client to server communication has moved to Editor's Draft state! The objective of moving to First Public Working Draft by mid-January has been set, and we are pushing hard towards it.
• Because of increased push on moving ActivityPump to this state, I have been added along with Jessica Tallon as co-editor on the specification.
• The "IndieWeb stack" (for lack of a better grouping) standards have all also advanced to Editors Draft status, including Webmention, Micropub, and Post Type Discovery. These standards are also on track for First Public Working Draft in mid-January.
• Amy Guy's work on a "convergence" standard has been renamed to Social Web Protocols and also (are you noticing a trend?) advanced to Editors Draft and is on track to First Public Working Draft in mid-January.
• I demoed ActiviPy, which went over really well. ActiviPy started out as a method of representing and working with ActivityStreams for Python, but I realized in developing it that since it was using JSON-JD with an implied context to handle ActivityStreams anyway, I could also extend it to support the Microformats vocabulary by using the JF2 context being developed for JF2. So I showed off a demo where I loaded ActivityStreams2 documents, demonstrated the method dispatch system ActiviPy uses (which is fairly interesting but I won't bore readers with it), but most excitingly, I loaded ActivityStreams and Microformats documents side by side in the same system and then converted them both to linked data! This got a fairly strong reaction from the room, since this was all three of the "directions" of achieving federation we've been working towards, with a real live demo of convergence! I was very proud to show this off.

Maybe most importantly was the "spirit of the room", and how much this has changed from prior meetings. This group was formed to work on some very challenging domains with the goal of bringing initial participants with some historically very differing backgrounds. But both the last face to face (which we mentioned in the last state of the goblin post) and this one have really done tremendous things towards propelling this group forward, and unbelievably, towards something that might actually be convergence (without requiring that).

Those who know enough historical detail of this space may be astounded to read that last sentence, but I believe it is true. In a sense, agreeing that convergence was not mandatory helped bring us towards a greater possibility of it. The agreement that ActivityStreams, Linked Data, and the "IndieWeb Stack" (for lack of a better term) were not required to work together, and that we could produce multiple deliverables, has eased that tension in the group and allowed us to work collaboratively. Everyone has worked hard to understand each other. But one person in particular has been doing a stand-up job of trying to bridge the cognitive gap and that person is Amy Guy. This can be seen immediately with her work on the Social Web Protocols document, but I believe she has done a great job otherwise in mapping the space.

So anyway, optimism can only bring you so far. There's a ton of work to be done in this space, and we'll be pressing hard.

See you at FOSDEM!

Are you going to FOSDEM? So will I! I'll be giving two talks in the Guile/Guix room, and I may be giving one more in another room, depending on acceptance or not. It would be good to meet other MediaGoblin community members or enthusiasts of MediaGoblin.

I should also say that I was really not sure if I could make FOSDEM originally, but a number of people very kindly donated to send me on my way. Thanks very kindly! I'll be sure to put it to maximum use for our community's sake. :)

We're also thinking of running some sort of dinner (or lunch?) for those who donate between now and FOSDEM to MediaGoblin, so hey, by the way, our donate page still works! ;)

Goblinoid updates

Goblinoid, you may remember, is the result of this year's Google Summer of Code, and is a neat MediaGoblin application for Android. Here's an update (thank you Laura Arjona and Dylan Jeffers for writing this up):

Dylan Jeffers, our GSoC 2015 student, is working in an Android app for GNU MediaGoblin.

Current features include viewing the recent activity feed, comment about media, upload photos from file... taking advantage of the Pump API.

The code is under heavy development, (repo here), and binaries for each release can be found (along with checksums) hosted at Goblin Refuge, a third party site kindly offered and maintained by SalmonLabs LLC, who also host the goblinrefuge.com MediaGoblin public instance.

Meanwhile we get the app in the popular free software repository F-Droid (it's taking a while because it's the first app in F-Droid built using the Python-based Kivy framework), we encourage everyone to test the app. The most up to date release is always available at https://files.goblinrefuge.com/download/Goblinoid/MediaGoblin-latest.apk) and report feedback in our IRC channel or the repository issue tracker).

We are very excited to make MediaGoblin part of your mobile life, and the Android app development is allowing improvements in MediaGoblin itself too (mainly fixes/improvements in the API-related code and the database and database migrations). Help us with your testing to improve that experience!

Wrapping up this year, onto next!

It's been a busy year. Here are some highlights:

• We took initiative on the challenges of user-centric hosting.
• We got out a new release of MediaGoblin and dug in for the hard work ahead of getting towards 1.0.
• We put your money to good use and funded Jessica to plow ahead with federation in MediaGoblin. This is well on its way, and we anticipate server to server federation to land in the next couple of months. Jessica just landed a massive overhaul to our database structure which was required to make this work, and that should make its way into the next release (coming soon).
• We became more engaged with the work to bring federation beyond MediaGoblin itself. Jessica and I both joined the W3C Social Working Group and have become co-editors on the ActivityPump specification and have devoted much time to advance these initiatives. This includes building tooling such as ActiviPy which will be critical for putting federation to real use within and outside of MediaGoblin.
• For the second year in a row, a major MediaGoblin member has received the O'Reilly Open Source Award largely (though not only)in recognition for their work on MediaGoblin. This year I (Chris Webber) received the award and last year Deb Nicholson received the award, which is rather incredible given the list of previous recipients of the award.
• Both Deb Nicholson and I (but especially Deb) have given talks at prominent conferences and have spoken on podcasts on issues of network freedom.
• We had a successful Summer of Code project resulting in Goblinoid, a free software client for MediaGoblin, which is Android and Replicant compatible. (It can run on GNU/Linux also!)
• And as we said, we were accepted as recipients for the Stripe Open Source Retreat! (Well, accepted this year, and the results of that should play out into 2016!)

That's a lot of stuff... what a year!

Well that's it from this update! See you in 2016, and happy hacking, fellow goblins!

---

State of the Goblin: September 2015

Quick announcement: I'm going to be making two appearances over the next week! First I'll be giving two talks on September 30th at Red Hat's Chicago office for the Chicago GNU/Linux User Group on Guix and Federation (both mentioned in this post)! Second I'll be attending the FSF 30th Birthday Party in Boston on October 3rd. If you're able to make it to either, do stop by and say hello... I'm expecting both to be a lot of fun!

Hello everyone! It's been a while since a comprehensive update of what's happening in MediaGoblin land. Despite the quiet, there is a lot to report, so let's get down to business and start reporting!

O'Reilly Award (again!)


Photo taken by Brandin Grams, CC BY 4.0, originally microblogged by Karen Sandler

First of all, something fun: I was fortunate enough to receive the O'Reilly Open Source Award! (Yes, I know about the terminology mismatch, we're free software people, but it's still a great honor, and I was presented the award under the description of my free software advocacy and GNU MediaGoblin work.) The other recipients of the award is quite the incredible group of people, and I'm honored to be listed among them. But here's what's really cool: you may remember that MediaGoblin co-founder Deb Nicholson won the O'Reilly Award last year. How's that as a vote of confidence in the things we're working on?



Anyway, if you want a more personal reflection, I wrote more on my personal blog!

Releases



So right, what about shipping software out the door? Well...



Since the crowdfunding campaign, we've gotten out two major releases, 0.7.0: Time Traveler's Delight and 0.8.0: A Gallery of Fine Creatures. I'm extremely proud of both of these releases! We have a lot more to do though on the road to 1.0, and we've been directly been putting the funds from the campaign to work to achieve that goal, so let's talk about that.

Putting your money to good work: Jessica and Federation



You may recall that we hired on the talented Jessica Tallon to get federation working in MediaGoblin. Jessica recently gave an update on the state of federation. Jessica is doing great work, though as expected, converting MediaGoblin to be a federated project has been no small task (knowing what a big task it was, hope that we could hire Jessica on to do this work was my #1 goal in the last campaign, in fact!). This decision has turned out to be absolutely the right one. Some of the best parts of the last two releases have been adopting the client to server Pump API. Federation has been MediaGoblin's goal since day 0, and Jessica is helping us to actually get there.

However (and now I'm going to do a pretty technical deep dive, so you can skip this paragraph if that isn't your thing), the most complicated aspect to making MediaGoblin into a federated project has had to do with updating the database to handle things while preserving data correctly for existing users. Why is this so complicated? A number of years ago we switched MediaGoblin over from using MongoDB to using either PostgreSQL or SQLite and while I believe this was absolutely the right decision, adding federation made the relational database system we had in place substantially trickier. For the more database-technically inclined, you can see that the ActivityPump API / Pump API require that any ActivityStream type object (in our case, that can be media or comments or even users) be referenceable by any type of activity. Furthermore, our existing comment system simply held that comments referenced media entries, whereas now comments can reference simply anything that is an ActivityStreams object. This means a large portion of our relations in our relational database needed an overhaul, and we needed a way to handle generic relations between tables. (The solution used is not unlike the "generic foreign key" implementation in Django.) There are more technical details on what has been done, but Jessica has been neck deep in this for months, but we believe we're finally on the home stretch, in which case Jessica can finally knock out server to server federation.

(I've thought that a whole post on database structure lessons learned may be a good blogpost of its own. One thing I'd note is that if jsonb had been an option when our current database design was put together, adopting that would have simplified things greatly, though it would require being PostgreSQL-only. But moving to that now would require a massive overhaul. If you're starting a new federated project from day 1, maybe keep that in mind!)

So the summary of all the federation stuff is: it's complex, but we're making good progress through Jessica's hard work. Expect more on this soon, and huge strides in the next release!

Federation and the W3C Social Working Group

So something Jessica and I have both been involved in over the last year is the W3C Social Working Group working towards official standards for federated web applications.


W3C Social Working Group March 2015 face to face meeting attendees. Jessica's holding the laptop on the left, and I'm right behind her.
Photo taken by Aaron Parecki, CC0 1.0, originally posted to the W3C wiki.

The federation protocol that MediaGoblin has been working towards until this point is primarily based on the Pump API, but this is really just a semi-formalization of the interface for the pump.io API. In the Social Working Group we are working towards defining a new standard, ActivityPump, which is based off of the ActivityStreams 2.0 standard. We're very excited with where this standard is going and feel it's a clean refinement over the Pump API we're already working with, while still keeping many of those same conventions.


W3C Social Working Group May 2015 face to face meeting attendees. I didn't make it to this one, but Jessica did, and did a stellar job representing MediaGoblin and ActivityPump!
Photo taken by Aaron Parecki, CC0 1.0, originally posted to the W3C wiki.

This has taken a lot of our time, but I believe it the results are worth it. Jessica and I have been attending weekly calls related to this standardization, and have thus far attended two face to face meetings at well. (More accurately, Jessica attended the second MediaGoblin-represented one without me, giving a kick-ass presentation on how ActivityPump works to the group! Go Jessica!)

As for my own personal work advancing this, I'll go into this a bit further on in this post!

Google Summer of Code result: Goblinoid!


Screenshot of Goblinoid, as it looks now!

Dylan Jeffers joined us for Google Summer of Code student this year work on a pretty cool project: a MediaGoblin client for Android or... really nearly anything... called Goblinoid! There's two really interesting features about Goblinoid: one, it's written in Kivy, a GUI toolkit which emulates the Android look and feel, but is actually can run nearly anywhere Python can run... making it quite portable, yet ideal for mobile computers!


Mockup of what Dylan would like Goblinoid to look like in the future!

So Goblinoid works... it could use more user testing and packaging, if you're interested in helping with that! But you can already upload images on the go via Goblinoid, and we expect more to come. Give it a go!

We've long been interested in having a client for MediaGoblin which makes use of the Pump API implementation we've been working on. Thank you Dylan for making that happen!

Infrastructure challenges

This has been a challenging year as in terms of supporting MediaGoblin's infrastructure. Spammers attacked both our wiki and bugtracker hard, at one point leaving me to take several weeks to fight issues and to try to find solutions. (Unfortunately, for the bugtracker, no great solutions have been found, and we are on a request-an-account basis... not a great situation to be in.)

Additionally, the primary Gitorious instance went down, where MediaGoblin's code was hosted along with many, many feature branches from contributors. The MediaGoblin community spent a while debating what we were going to do. A move to GitHub was tempting but is not an option; that's exactly the opposite of the type of world we want to build. Not everyone in MediaGoblin's community is comfortable with GitLab, and their primary instance is running a proprietary version. There are some other communities hosting things like Notabug who seem to be run by great people, but we run the risk of running into these same problems all over again... though so we did with most of these other solutions. We could self-host, but there is very little time for extra server maintenance burdens right now. So we moved our git repository over to Savannah and I'm glad to have hosting there by people I trust, though I do also feel that contributors may expect more modern hosting facilities, but we don't have time to run them ourselves.

The Gitorious shutdown came around the time of great exhaustion of already dealing with issues related to the bugtracker and left me feeling very burnt out. But it also lead to a great amount of reflection... who am I to feel frustrated with? The Gitorious people graciously hosted our software repositories for some time, and I was not willing to run an instance of my own. Why not? Well, one problem is that if you run your own instance of a free software web application that isn't federated, you're working in Yet Another Semi-Free Micro-Silo (TM). But let's face it, that's not the real major problem... looking at the frustrations we've had with Trac, the answer is obvious: running free network services is a huge maintenance burden.

But wait... yes, you may be catching a whiff of irony here... if I'm not willing to run a free software web application because I don't want to take on the maintenance burden of someone else's software, how can I ever expect MediaGoblin to gain adoption?

And here's where I come to a tough, but I think necessary, conclusion: there's simply no way for MediaGoblin to succeed if the world of deployment stays where it's at. Something must be done. But what?

Research into deployment and federation

Partly affected by the above, summer came around, and I had a talk with MediaGoblin contributors in our monthly meetings. I wanted to take a sabbatical... a sabbatical where I was on break from "direct" MediaGoblin things, so I can advance things that affect MediaGoblin greatly indirectly. (My spouse and I were also going through a big move so this was a good time to do it, I figured.) I wanted to do research into two things: deployment and federation.

Framing the deployment side of things, Deb Nicholson and I gave a talk that kicked off "userops", a term that I still feel accurately captures what we're trying to do (and a talk which I still believe accurately summarizes the challenges we're facing). Within this context I began exploring options of what can be done to improve deployability.

The directions I've explored, and why I've come to the conclusions I have, are a series of blogposts of their own (if you're interested, I suggest subscribing to the userops mailing list, where I will be posting more as I go). The short of it is that I laid out a set of requirements to achieve easy and maintainable deployments, attempted to explore them with the most popular current tools (Ansible, Puppet, Salt, Docker, etc), and found that it is not possible to build both easy and maintainable systems on top of them based on what I believe users need. This lead me eventually down the path towards Guix, a package manager (and with GuixSD, also a distro) and soon to be deployment system which I am very confident has the potential to solve many of the challenges which make deploying and maintaining systems too exhausting a task for the average user. The software is nowhere near being "easy" at present (I think it's very telling to say that it's "the Emacs of package managers / distributions"... one can extrapolate on that in many ways, most of which are correct, except for the Emacs-haters kind... lay off, Emacs haters!) but I think has the potential to become so. An easy to use web user interface has already been demonstrated, and I believe the foundations are good for building a complete and easy to use system for everyday users. But again, to go into detail beyond what has already been said is something that will be explored elsewhere.

There is a more pressing need for me to have explored deployment at present as well... though I want to explore deployability for the sake of other users, I also need to explore deployability for my own sake... particularly because we promised that we will provide premium hosting in the last campaign we ran! Yes, in case you have been wondering about it, I have not forgotten about this promise. How to fulfill this promise without being crushed by the maintenance burden of hosting? I came to realize that there was every risk that I would spend all my time supporting and maintaining servers that were running MediaGoblin and I would not have the opportunity to any longer be a steward of MediaGoblin itself... which could lead to failure. So figuring out a better path forward on hosting has become a necessity. I'll explore what this means further in the next section, but first, I should say a word on federation and what my sabbatical lead to on this front.

As I have said, Jessica and I have been involved in the W3C Social Working Group. Part of the activities of the group have been the definition of the ActivityStreams 2.0 and Pump API standards.

I set out to explore this a bit more deeply under a repository with the joking and interim name of activitystuff (the Social Working Group is using GitHub for its work, so I'm making an exception there). Along with some other projects, this has contributed to a deeper understanding of how federation should work, which is something useful to take back to MediaGoblin. There are some potentially useful things in that repository (including a partially complete implementation of the JSON-LD API), and it may turn into a full implementation of ActivityPump, though its original and primary goal was for exploration, a purpose it has served well.

One major event relating to federation has just occurred over the last week and bears note here: a number of Pump.IO community members and myself are now working with Evan Prodromou (who has near-certainly contributed more to the federated web space than anyone else alive) to transition the project from being a project of primarily stewardship under Evan to one of community stewardship and governance. I posted a summary of a recent meeting, and (MediaGoblin contributor!) Laura Arjona put together a community document. In sum, Pump.IO could use your help if you're interested.

So anyway, it's been a busy last couple of months! But it's time to return to MediaGoblin-ville, so...

What's next?

Well, that's a whole lot of text above, so how about a bulleted list next? I hear those are easier on the eyes.

• I'm swooping back into MediaGoblin territory starting next month. However, my initial focus will be on getting MediaGoblin to a deployable point for myself, particularly forward-looking towards making premium hosting feasible. Expect more soon!
• Jessica is working on MediaGoblin federation. We expect the massive database changes she has been working on to tidy up in the next few weeks, and if all is well, we'll have server to server federation basics in place by the end of the year.
• Work towards 0.9.0 will proceed, unless 1.0 lands first! Expect major infrastructure improvements in 0.9.0, and federation to land in 1.0.

Those are all great things, and I think we are indeed on track towards our goals, slowly but surely.

There is a more rough side to this: we've allocated nearly all of the funds from the last campaign towards paying for Jessica's work on MediaGoblin, and she's devoted enough to the project that she's been working well, well below market rate. (And aside from some travel reimbursements, I have not taken money personally from the last campaign.) I'll post a financial transparency report soon, but the short of it is: the MediaGoblin funds are running low. Even with Jessica generously working for such a relatively low amount of money, we won't be able to pay Jessica to work on MediaGoblin much longer, given our present finances.

But we aren't giving up! The goals of MediaGoblin and of federating the web are just too important, and so onward we press, into the future! Though MediaGoblin is an ambitious project, I am confident we can achieve the things we have set out to do, but we are constrained by limited resources and time.

Appreciate what we are doing? Want to help us in our quest to bring network freedoms to everyone? You can help!

Donate...

• The simplest way to help? Donate! Though the official campaign is over, you can still donate to MediaGoblin through the FSF!
• Jessica and I have been contracting so we can pay the bills (doing this allowed us to "stretch out" the amount of time Jessica could spend on the project... useful with all that W3C stuff in progress!). Unfortunately, while the people we have been working with are great, the contract we've been working on is coming to an end. Are you looking for contractors or part time workers who are capable of developing high quality free software and providing community leadership? Jessica and I are both interested in working in such cases... feel free to email me at: cwebber AT dustycloud DOT org

We're committed to making a better, decentralized web. I hope this piece cleared up where we're at and where we're going. I believe we've got exciting times ahead... till next time, goblins!

---

The state of Federation

It's been a long time since there has been any news on the state of federation, so here's an update on where MediaGoblin's at and some technical aspects of federation. We've been working with the W3C Social Working Group to define the future of federation, and part of my work there has been to work on the ActivityPump standard. There's more to say on that and why we're investing time there, but this blogpost will mostly be about MediaGoblin and federation from a technical perspective.

Over the last year I've been building up the foundations of federation, creating the necessary APIs and infrastructure needed. I am currently finishing the last of that, modifying the models to handle federated data. This is a challenge as the models were designed without federation in mind. There is a lot of new data that is being stored as well as remote and local versions of the models.

Once I've finished working on the models I'll be starting the work on actually getting the media and comments federating across instances.

The core of the federation and APIs are "Activities", these are created all the time right now in 0.8.0. Every time someone posts media, comments, tags or updates anything. These serve two main purposes:

• Command language: instructing the server to do something
• Log: telling clients and other servers what has been done

The activities at their core at subject predicate object, for example:

Chris posts an image

The activity also usually contains the audience targeting data (i.e. who it's for). These are represented in JSON with elements being nested as other JSON objects, lets take a look:

{
    "id": "https://mediagoblin.com/api/activity/1"
    "verb": "post",
    "objectType": "activity",
    "actor": {
        "id": "acct:cwebber@mediagoblin.com",
        "displayName": "Christopher Allan Webber",
        "objectType": "person"
    },
    "object": {
        "id": "https://mediagoblin.com/api/image/1",
        "objectType": "image",
        "url": "https://mediagoblin.com/u/cwebber/m/mediagoblin-shirt-close-up-and-badge/",
        "fullImage": "https://mediagoblin.com/mgoblin_media/media_entries/814/IMG_1011_modified.jpg"
    },
    "to": [
        {
            "id": "https://mediagoblin.com/api/user/cwebber/followers",
            "objectType": "collection"
        }
    ]
}

This might look overwhelming but it's the same basic subject predicate object sentence above. Chris (the actor) is posting (the verb) an image (the object), it also contains some extra information like IDs so they can be referenced and the audience which is a a collection of people that follow Chris.

Once we have this activity, what actually happens, you ask? Each user has an inbox and outbox like you do with email. Step by step walkthrough of what would happen when someone posts an image from a desktop or mobile client:

1 Client to server

back a preliminary object to use in the API.

1. The client creates an activity that at minimum would look like this:

{
    "verb": "post",
    "objectType": "activity",
    "object": {
        "id": "https://mediagoblin.com/api/image/1",
        "objectType": "image",
    },
    "to": [
        {
            "id": "https://mediagoblin.com/api/user/cwebber/followers",
            "objectType": "collection"
        }
    ]
}

2. This activity is then posted to the client's inbox.
3. The server receives the activity, creates an ID, and attaches the actor and other metadata, then saves it.

2 Federation!

where we're all on our own servers. This is what the server (once it's finished) will do once it gets the client's request:

1. The server looks for whom it's sent to (the "to" property) to get a list of people that the object has to be sent to.
2. The activity is then sent to each persons inbox.

As one of Chris' followers you can now open up the website or any client and see Chris' image. Commenting on the image will produce an activity which is sent to Chris, who subsequently sends it as an update to everyone else.

Phew! we're done, that's how the API and federation works in MediaGoblin. We've successfully posted an image from a client and had it sent around and hopefully you're able to see how a comment or something else would work in this system too.

This is just a high level overview, there are many technical problems such as access control (who's able to view content), verification of objects (checking what we've been sent is correct), etc. Federation doesn't come easy but we think it's well worth it.

---

MediaGoblin 0.8.0: A Gallery of Fine Creatures



We're excited to announce that MediaGoblin 0.8.0, "A Gallery of Fine Creatures", has been released! The biggest news is that the client to server API (making use of the future federation API) is much improved! That means that users no longer have to depend on a browser to access MediaGoblin. You can access and post to your MediaGoblin instance via any of several Pump.io compatible clients, like Pumpa and Dianara (or write your own using PyPump)! The grand goal is a generic (and ubiquitous) client protocol that will work with lots of different served applications that use the pump standard. Eventually, any Pump API compatible client will essentially be a MediaGoblin client and a Pump.IO client. We expect more client types to be added very soon!

Part of the process of world domination via federation means that we're now able to support serving content to multiple client types through a single protocol.

For example, here's a user uploading an image using the Pump.IO client, Pumpa:



And here's that same image, now uploaded to MediaGoblin!



In a nutshell, the client to server part of the API/federation equation is working. We are still working on server to server federation that will enable us to share comments, tagging and all the other things that can happen to your shared content on someone else's server that you may (or may not) want hear about.

By the way, if you're using MediaGoblin with Apache (rather than Nginx or some other setup), you'll need to add "WSGIPassAuthorization On" to your config or the API won't work. You can look at this wiki page for reference.

Speaking of updating and getting with the times, we are officially offering preliminary support for Python 3. Most of our features work without Python 2 installed, so welcome aboard futurists! As always, if you spot something we missed, we'd love to hear from you via our bugtracker, mailing list or IRC channel.

Also on the upgrade list for this release, GStreamer! We're now using version 1.0 which adds a nice new thumbnailer and includes much improved video transcoding support. If you didn't get thumbnails before, it wasn't you. We encourage you to try again and see if it works for you now!


Thumbnails from Venom's Lab, licensed under CC BY-SA 3.0

Obviously, the future demands cleaner packaging. Configure and make support is now the default... welcome to fewer steps for installing your MediaGoblin instance, which will be critical as we build packages for Fedora, Debian and any other distro that wants to help its users host the federated future of the web.

We've also switched away from Transifex, which had become proprietary (boo!) to an instance of Pootle that many of our fellow GNU projects are now using for translations. If you've been looking for a fully free translation tool, Pootle may be just the thing you've been looking for!

And finally, we fixed the footer. It is now forced to the bottom of page, instead of floating in the middle of short pages which everyone agrees was sub-optimal.

Next up is server to server federation. This is what Jessica Tallon has been working on full-tilt -- thanks to everyone who pitched in on the MediaGoblin campaign. (Though the campaign is over, you can still donate!) Both Chris and Jessica have been participating in the W3C Social Working Group on a federation standard. (As well as a general purpose client to server API... you better believe the increased client support coming out of this release is related!) The design is very closely related to the existing Pump API spec we've been using. You can read the current draft of the standard here. We've got more news on the way, including on that federation front. Expect more news soon!

Thanks to everyone who is helping us make the future of the web happen everywhere! This release wouldn't have happened without the help of these people: Alon Levy, Asheesh Laroia, Andrew Browning, Berker Peksag, Boris Bobrov, Christopher Allan Webber, Deb Nicholson, Ineiev, Jaakko Luttinen, Jakob Kramer, Jeremy Pope, Jessica Tallon, Jim Campbell, Laura Arjona, Meg Ford, Rodrigo Rodrigues da Silva, and Ben Sturmfels. You all rock!

Want to put in your own help towards the future of federated, awesome media publishing? Join us! Visit us in IRC (#mediagoblin on freenode.net) or sign up for regular updates on our mailing list Got ideas or questions about our work? Email us at press AT mediagoblin DOT org -- we look forward to hearing from you!

---

Userops: Deployment for the People

Deb Nicholson and I both recently gave a talk at FOSDEM 2015 called "Can Distros Make the Link?" (A recording is here, and my slides are here, hit "s" for speaker notes or read the org-mode source if you prefer.) The main purpose of the talk was that packaging libre network services/applications for distros is important, but distros in their present forms aren't really enough to solve the deployability problems and pains that anyone trying to run their own libre servers knows. I had a bit of a worry that this thesis would upset part of the audience (it was in the distros room, after all) but it turns out that everyone seemed to agree and be on board.

Many audience members even encouraged us that this conversation needed to continue beyond FOSDEM, and there was discussion of hosting a mailing list to continue the conversation. As usual, everyone had various ideas of where to host it, but the audience seemed to feel that MediaGoblin's servers were fairly neutral ground, so we announced that we would put up a mailing list and announce it here when we got the chance.

It's a bit delayed, but I'm happy to announce the launch of the userops mailing list! If you're interested in talking about making deployment easier for every-day users, please consider joining the conversation there. (Oh, and we also have an IRC channel: join #userops on irc.freenode.net, if you're the IRC type!)

Why the name "userops"? As you may have guessed, this is a pun on the term "devops"; the idea is that we also care about configuration management and deployability, but we aim for a different audience. Devops, as the name implies, focuses on liberating developers in the world of deployment, particularly developers who have to deploy a large number of machines for $LARGE_CORPORATION at their job. Userops, on the other hand, aims at liberating users in the world of deployment. You shouldn't have to be a developer to take advantage of network freedoms and run network-oriented free software. After all, the free software world generally agrees that it makes sense that users of desktop software should not have to be developers, and that "user freedom" takes priority over "developer freedom"... the freedom of $LARGE_CORPORATION, while not something we object to, is not really our primary concern. (Though of course, if we build solutions that are good enough for end-users, corporations will probably adopt them, and that is fine! It just isn't our focus.)

(Oh, and in case you stumble upon it, "userops" was originally a name I had for one of my personal projects experimenting with deployability, but a friend of mine convinced me that the term was too useful to be constrained to one particular piece of software, so I've renamed that project! Everyone is now free to use the term "userops" to refer to the vision described above.)

We believe that "userops" is now more important than ever. These days, it is not just enough to use free software network services, one must have the ability to deploy and make use of that software. (For many of this, the timeliness and urgency of this is seen with the turmoil for many free software developers figuring out where to go now that the hosted version of Gitorious is being shut down.) And as you may have guessed, we're well aware of how true this is not of just "all that other libre network services"; MediaGoblin requires quite a bit of technical skills and resources to run. We'd like to improve that, but we think there are some real challenges that are beyond what MediaGoblin can do as MediaGoblin itself: things need to happen on another layer (or layers) too. Hence "userops"!

If this is something you likewise care about, and especially if it's something you're working on or thinking about (or would like to), consider joining the conversation!

---

MediaGoblin 0.7.1 released

MediaGoblin 0.7.1 has been released! This is a bugfix release building on MediaGoblin 0.7.0.

Upgrading is highly encouraged to those running PostgreSQL databases especially. There were some issues in the previous release that lead to users of PostgreSQL to see random errors. We had some problems related to a couple of (non-critical) features not handling transactions well... these have been disabled for this release, but will likely be re-enabled by 0.8.0's release.

Thanks to everyone who made this release possible: Andrew Browning, Christopher Allan Webber, Jessica Tallon, Low Kian Seong, Matt Molyneaux, and Odin Hørthe Omdal. Thanks so much!

Please see the release notes for details. Happy goblin'ing!

---

Deb Nicholson receives O'Reilly Open Source Award

Those of you who follow MediaGoblin closely likely know of Deb Nicholson, our community manager. This post is a bit late, but nonetheless, I wanted to share something exciting that happened:


Deb Nicholson receiving the O'Reilly Open Source Award.
Photo by Bryan Smith, released under CC BY-SA 3.0 Unported.

That's right! Deb Nicholson won the O'Reilly Open Source Award, one of the most recognized awards in the FLOSS world, for her work on GNU MediaGoblin and OpenHatch. Here's the text of Deb's nomination:

Deb Nicholson has been a sparkplug for Linux and FOSS advocacy and she can best be described as the traffic cop at the intersection of technology and social justice. A free speech advocate, economic justice organizer and civil liberties defender for years, Deb became involved in the free software movement. She is the Community Outreach Director at the Open Invention Network and the Community Manager at MediaGoblin. She also serves on the board at Open Hatch, a non-profit dedicated to matching prospective free software contributors with communities, tools and education.

...as well as the message from the award announcement itself:

Deb Nicholson works at the intersection of technology and social justice. She became involved in the free software movement about five years ago when she started working for the Free Software Foundation. She is currently the Community Outreach Director for the Open Invention Network and in her spare time, she serves on the board of OpenHatch. Congratulations Deb!


Photo by Bryan Smith, released under CC BY-SA 3.0 Unported.

Deb and I have known each other from when she used to work at the Free Software Foundation and we became friends over time. I've always been impressed with the tireless work Deb has put into making the world of free software a better and friendlier place for everyone, as well as her extensive knowledge gained from a long history of social justice work. Early on in MediaGoblin starting up as a project, I talked to Deb about it and asked if she'd be interested in being involved. I'm glad I did... Deb has brought much to the project and I constantly lean on her skills in writing, her sharp wit, and her clear and regular guidance on how to build MediaGoblin into a better community. MediaGoblin wouldn't be the same place it is today without Deb working with us.

Deb, thanks for your work on MediaGoblin, OpenHatch, and so many other things... and for being a great friend to myself and many others in the MediaGoblin community! We're glad to have you here!

---

MediaGoblin 0.7.0: Time Traveler's Delight



Welcome to MediaGoblin 0.7.0: Time Traveler's Delight! It's been longer than usual for our releases, but we assure you this is because we've been traveling back and forth across the timeline picking up cool technology that spans a wide spectrum of space and time. But our time-boat has finally come into the harbor. Get ready... we've got a lot of cargo to unpack!

You may remember the work we are doing towards federation, and even the demo we showed earlier of that progress.

Well we're excited to announce that the first piece towards MediaGoblin federation has landed! We don't have server-to-server federation working yet, but we do have the first parts of the Pump API in place: you can now use the Pump API as a media upload API! Are you a python developer? Starting a client couldn't be easier now, using PyPump! We also have a whole new section of our docs about the Pump API. There's of course more Pump related things to come in future releases, but we're excited to be well on our way!


Our new theme Sandy 70s Speedboat looks great on galleries...

Sailing into this release is an excellent new theme from Jeremy Pope: Sandy 70s Speedboat! This retro-styled, light colored theme has just enough frills to make your site look good while emphasizing the real stuff you want to show off... your media!


... and on individual media, too!

MediaGoblin is now using the skeleton CSS system, making it more responsive. MediaGoblin sites now adaptively fit better into a variety of resolutions, including mobile phones, across the board. (Responsive design is the thing all the cool kids are into these days right?) Now MediaGoblin is much nicer to look at on the go!


Now more responsive!

We also have a new blogging media type. However, it's very experimental and could use more testing and careful code review... but if you're interested in testing and helping out in this area, check it out!

In addition, we have a number of features that have come in thanks to work from a grant to improve MediaGoblin in use with galleries, libraries, archival institutions, and museums. The first of these features is something people have long wanted: the ability for site administrators/curators to "feature" media to appear on the frontpage of a site.

We also now have a tool for command line bulk uploading that has come in through this grant work. Do you already have a set of media and you need to pull into a MediaGoblin instance? You can now use the command line bulk upload tool to automate pulling in that media, including setting metadata.



Wait, metadata? What do we mean by that? Well, what if you want to store some extra information about some work? (What year was this painting done in? If the author was different than the uploader, who was the original author? And many other things!) Now you can associate this information easily with media that you are uploading. With the appropriate plugin enabled, this information is viewable to the user... but it's also machine readable. Now even robots can appreciate the cultural works on your MediaGoblin site!

For site administrators, we also have two new subcommands: "deletemedia" and "deleteusers". Whew! Now you can get that cruft that shouldn't be there off your site in an automated manner!

There are many other fixes and improvements in this release... too many to detail! But some highlights are: the long-hated "video thumbnails not generating" bug is fixed, many improvements to translations, fixes to the PDF media type, new default permissions options for the config file, new template hooks for plugins, and much, much more!

Whew... that sure is a lot! It's good to see that our time travel madness has paid off in a bounty of fixes and improvements. In the meanwhile, this release was a huge group effort (as always!) so let's thank our contributors for all their hard work: Aditi Mittal, Aleksej Serdjukov, Alon Levy, Amirouche Boubekki, Andrew Browning, Berker Peksag, Beuc, Boris Bobrov, Brett Smith, Christopher Allan Webber, Deb Nicholson, Elrond (of Samba TNG), Jessica Tallon, Jiyda Mint Moussa, Jeremy Pope, Laura Arjona Reina, Loïc Le Ninan, Matt Molyneaux, Natalie Foust-Pilcher, Odin Hørthe Omdal, Rodney Ewing, Rodrigo Rodrigues da Silva, Sergio Durigan Junior, Sebastian Spaeth, Sebastian Hugentobler, and Tryggvi Björgvinsson. Thanks so much everyone... we really couldn't do it without you!

Stay tuned for more. We've got more cargo that's shipping its way on in for the next release... we'd better get back to work! In the meanwhile, enjoy this release and be sure to check the release notes. And if you're interested in joining our crew, we'd love to have you on board, so please do join us!

Happy travels, everyone!

Update: Are you upgrading from a previous version of GNU MediaGoblin? The release notes left out a step (now corrected)... you should also run the command "git submodule init && git submodule update". Otherwise you'll be missing out on the "skeleton" CSS framework and things will look really weird! Not to mention the sandy 70s speedboat theme! If you're doing a new install, this won't be a problem.

---

Welcome Jessica Tallon, MediaGoblin's second full time hire



I'm excited to announce that MediaGoblin has hired its second full-time programmer: Jessica Tallon! Those of you who follow MediaGoblin closely may recognize that name: Jessica joined us as part of our Outreach Program for Women participation last year (she wrote about her experiences with the program on this blog). Jessica has been working on federation support in the project.



Since our crowdfunding campaign was a massive success, reaching its second milestone, that was enough funds to bring Jessica on full time. (You may be wondering: what about me (Chris Webber) then? Am I still full time on the project? The answer is "nearly"; I am full time without pay for now, but will be picking up a small amount of contracting to cover the bills so I will be at the status "nearly full time"... thus MediaGoblin will have mostly two full time people on staff at the moment!) You might notice that the number of dedicated resources on the project corresponds with the "number of goblins unlocked". That's no coincidence... we made no official announcements because we couldn't be sure until everything was arranged, but the number of goblins on that page was the number of dedicated resources we hoped to pull in. Luckily, we were able to make an arrangement and pull in Jessica. That's great news for MediaGoblin!

Already, the results are showing themselves. Jessica only joined a couple of weeks ago and already she has landed the first major milestone for federation (not federation itself, but the upload API... it'll be in the next release, so more soon!) and has been helping get 0.7.0 out the door. The results speak for themselves!

But even though the results speak so clearly, there's nothing like hearing right from the source! As such, Deb Nicholson was kind enough to interview Jessica... so without further ado, here is their conversation below!

---

Deb: You started working on GNU MediaGoblin through an Outreach Program for Women internship. What initially drew you to MediaGoblin, instead of one of the other fine projects that offer paid summer internships?

Jessica: MediaGoblin was a perfect fit for both me and what I wanted to work on. The welcoming community was something which was immediately apparent, I wanted to work on a project which was going to be welcoming and for me to feel like I could continue after OPW had concluded. I also wanted a project which would provide me with any help I might need while getting to grips with the code. It was clear that MediaGoblin was a perfect fit for me in that respect.

Deb: You spent last summer re-writing the PyPump library. Are projects besides GNU MediaGoblin using your work?

Jessica: I've seen several projects which are using PyPump as well as interest in helping develop the library itself. Spigot is a program written against PyPump that allows posting to RSS feeds to pump.io. There is also PumpMigrate which migrates data from one account to another, this could be really useful both on pump.io and MediaGoblin to allow someone to move their account to a different instance.

Deb: You'll be working on GNU MediaGoblin's federation branch for the next year. What do you think the biggest or most challenging part is going to be?

Jessica: I think that there will be two big and difficult aspects I will encounter over the next year. The first will be across instance subscriptions which would allow you to subscribe to a collection or user and have updates to those be visible on your instance even though they exist and have been uploaded to another instance.

The second and possibly the most difficult but also most rewarding part will be the sharing of media. This will allow you to share some media uploaded from a MediaGoblin instance via your own MediaGoblin account possibly on another server or even a pump.io instance. Permissions on how users will be able to interact with the media by commenting, favoriting or re-sharing will make the problem interesting but brings challenges.

Deb: What do you think GNU MediaGoblin will look like in 3 years time?

Jessica: This is a very difficult, over the last year that I've been apart of the project I've seen so much happen. I think within 3 years we'll have mature federation support which will make running your own instance easy without isolating yourself from the social aspects of sharing your media. There's also lots I see happening from others in the project for easier deployability which will hopefully make deploying a MediaGoblin instance much more achievable for a lot more people.

I am so excited to see what unexpected changes occur over the next 3 years. There are so many contributors and it's such a wonderful community and project that I know I'm always going to be blown away by what can be achieved by everyone involved.

---

MediaGoblin campaign a massive success! What's next?



So this happened. To say "we are floored with the success of the campaign and the community support we've received" would be a huge understatement. We made the call to arms to fight for federation and privacy features on the web, and you answered.

Like an adventure movie, the stakes were high, at times we weren't sure we were going to make it, but there was excitement abound and in the end, we were victorious. We surpassed not one but two milestones and met a 10k matching grant (the final thing putting us over the top of that last one, by the way, was cashing in our Bitcoins). We gave a complete and total financial transparency breakdown, and you responded with appreciation. We came to the last week of the campaign at 67% of the way to the second milestone... still a ways to go... could we do it? And we did! Or we should say, YOU did. You made this possible.

But that's hardly the end of our adventure... the fundraising campaign is over, but the most the most exciting times are ahead of us. Let's look at just what you all have succeeded in unlocking here:




Wow, feature-wise that is a lot of stuff! From federation, to private media sharing, to PyPump development, to podcast support... that's a lot of features alone. And I'm sure we're all excited about the 1.0 release ahead! You've funded all of this, which means we are getting to work on it. We'll be sure to be making updates as we move forward with implementing it all.

There are more exciting updates on the way, also. Two of the things unlocked we'll be making more announcements over the next couple of weeks, but we are preparing for them to make sure we can pull them off in a way that's best for everyone. The first of these is the "donor chosen feature". We're going to give donors a list of significant features to choose from, but we want to make sure we have a list of things that's both feasible and interesting to pull off.

The second is the premium hosting reward. This got unlocked as a reward option just barely a day before the campaign ended. We thought it would be pretty crazy (and unfair) to offer this option right before the end of the campaign and then only give people a few hours to select them, so here's what we're doing instead: the main rewards from the campaign are now closed down (you can still donate if you like), but very shortly we're going to open things up again, but with only premium hosting as an option. We think this is the best route for everyone: it will allow us to properly plan out the reward in a way that works well, and it will give people a chance to actually opt in to the reward.

So, that's plenty to keep looking forward to! Keep your eyes here for more updates on the above!

Last of all, what about all the rewards you all ordered? The plan is that we will begin delivering on those over the summer in order to maximize reward delivery with actual project productivity. My spouse Morgan Lemmer-Webber will (very kindly) be assisting with reward fulfillment over the summer, in-between her research projects. We realize some of you may be eager to get your rewards, but hopefully you'll be happy to realize this means we'll be able to optimize time between getting those out and getting work done on the real stuff you funded: MediaGoblin itself!

So, that's a lot to look forward to. We hope you're as excited as we are as we step into this new era of MediaGoblin development. And there are surely more exciting things to announce along the way.

In the meanwhile, it's worth repeating: thank you to everyone who supported the campaign. We're tackling some ambitious stuff here in MediaGoblin-land with the end-goal being nothing short of paving the way for a better internet. Thanks for helping us make that possible, and joining us on this journey. We couldn't do it without you.

---

« Page 2 / 8 »