MediaGoblin 0.12.0: Potions

    Potions
    "Potions" by Charlotte Koch (reproduced with permission)

    Happy Software Freedom Day! Today we're pleased to announce the release of MediaGoblin 0.12.0. See the release notes for full details and upgrading instructions.

    This release resolves two significant issues in the Celery backend media processing. The first was causing processed media to be marked as failed and the second was inhibiting useful error messages. We've also resolved installation issues caused by deprecated upstream code in the Werkzeug and jsonschema libraries.

    We've added provisional gmg serve and gmg celery commands to simplify deployment. These commands may change in the future and are not yet recommended in the deployment documentation. If your deployment is already running smoothly, there's no reason to switch at this stage.

    Thanks to Rodrigo Martins, Marco Pessotto and Jgart for their contributions in this release. Thanks again to Charlotte for allowing us to feature her artwork.

    To join us and help improve MediaGoblin, please visit our getting involved page.


    MediaGoblin 0.11.0: Punky Magmalian

    Punky Magmalian Portrait
    "Punky Magmalian Portrait" by Charlotte Koch (reproduced with permission)

    This release of MediaGoblin removes support for Python 2 and marks the end of five years of concurrent support for both Python 2 and Python 3. That's a major achievement, so congratulations to everyone who contributed.

    As a bonus, the release also includes a completely rewritten replacement for the previously Python 2-only audio spectrograms feature. Thank you to Fernando Gutierrez for this contribution; it's a significant piece of work. The release also includes contributions from Jeremy Bowman, Jesús Eduardo Estupiñan Medina, Charlotte Koch, Olivier Mehani, milquetoast and Ben Sturmfels.

    On reflection, the Python 3 transition has been bitter-sweet. Without doubt, Python 3 is both technically and ergonomically superior to Python 2, but I don't think anyone in the Python community realised quite how long or how much work the transition would be. It's been an especially challenging time for small projects like MediaGoblin and for anyone supporting a significant Python codebase on limited resources. Looking forward though, ending support for Python 2 significantly simplifies the maintenance of the project, making it easier to add new features and prevent bugs. We're really excited about this! Although we're not bug-free yet, as of this release our test suite is passing 100% and we'll be continuing with further quality improvements.

    We would like to take this opportunity to acknowledge Boris Bobrov, project maintainer and long-time contributor. Boris recently retired from his maintainer role which he began in 2016. Thanks Boris for all your hard work, contributions and leadership, we really appreciate it!

    This release is named Punky Magmalian after a character created by artist and MediaGoblin contributor Charlotte Koch. Take a look at Charlotte's other work on her MediaGoblin site.

    For more details on this release and on upgrading see the release notes. To join us and help improve MediaGoblin, please visit our getting involved page.


    MediaGoblin 0.10.0 released

    We're pleased to announce the release of MediaGoblin 0.10.0!

    It's been a while between releases for MediaGoblin, but work has continued steadily. Highlights of this release include a new plugin for displaying video subtitles and support for transcoding and displaying video in multiple resolutions. There have also been a large number of smaller improvements and bug fixes which are listed in the release notes.

    After enabling the new subtitles plugin, you can upload and edit captions for your videos. Multiple subtitle tracks are supported, such as for different languages. This feature was added by Saksham Agrawal during Google Summer of Code 2016 and mentored by Boris Bobrov. The feature has been available for some time on the master branch, but it definitely deserves a mention for this release.

    A screenshot showing MediaGoblin with subtitles shown on a video
    A video with subtitles added

    Videos are now automatically transcoded at various video qualities such as 360p, 480p and 720p. You can choose your preferred quality while watching the video. This feature was added by Vijeth Aradhya during Google Summer of Code 2017 and mentored by Boris Bobrov. Again this feature has been available for some time on master, but is also worthy of a mention.

    A screenshot showing the video quality selector in MediaGoblin
    Selecting a video quality

    For details on installing MediaGoblin, see Deploying MediaGoblin and for tips on upgrading, see the release notes. To join us and help improve MediaGoblin, please visit our getting involved page!


    We're still here!

    | tags: mediagoblin

    Hello Goblin-Lovers! [tap tap] Is this thing still on? ... Great! Well, we've had a few polite questions as to what's happening in MediaGoblin-land, given our last blog post was a few years back. Let's talk about that.

    While development on MediaGoblin has slowed over the last few years, work has continued steadily, with significant improvements such as multi-resolution video (Vijeth Aradhya), video subtitles (Saksham) and a bunch of minor improvements and bug-fixes. Like most community-driven free software projects, progress only happens when people show up and make it happen. See below for a list of the wonderful people who have contributed over the last few years. Thank you all very much!

    In recent years, Chris Lemmer Webber has stepped back from the role of much-loved project leader to focus on ActivityPub and the standardisation of federated social networking protocols. That process was a lot of work but ultimately successful with ActivityPub becoming a W3C recommendation in 2018 and going on to be adopted by a range of social networking platforms. Congratulations to Chris, Jessica and the other authors on the success of ActivityPub! In particular though, we would like to express our gratitude for Chris's charismatic leadership, community organising and publicity work on MediaGoblin, not to mention the coding and artwork contributions. Thanks Chris!

    During this time Andrew Browning, Boris Bobrov and Simon Fondrie-Teitler have led the MediaGoblin project, supported the infrastructure and worked with numerous new contributors to add new features and bug-fixes. More recently, I've stepped up to support them and deal with some of the project admin. I've also been working an exciting pilot project here in Australia using MediaGoblin to publish culturally significant media in remote indigenous communities.

    Back in February we held the first community meeting in quite a while. We met via a Mumble audio-conference and discussed short-term project needs including problems with the issue tracker, urgent/blocking bugs, a release, a bug squashing party, and the need for this blog post. Next meeting we'll be diving into some of the longer-term strategy. Keep an eye on the mailing list for the announcement and please join us.

    Based on that meeting, our current short-term priorities are:

    1. Improve/replace the issue tracker. There was general agreement that our current issue tracker, Trac, is discouraging new contributions. Firstly, registrations and notifications were not working properly. Secondly, the process of submitting code is more complicated than other modern collaboration tools. Our friends at FSF are currently working to select a new collaboration tool, so we'll look forward to evaluating their recommendation when it is announced. In the short-term, we've fixed the registration and notification problems with Trac to keep us going.

    2. Make a minor release. A release is an important opportunity to highlight the work that's been done over the last few years such as the multi-resolution video and subtitles I mentioned, as well as important fixes such as to audio upload in Python 3. This will likely also be our last Python 2-compatible release. Many of MediaGoblin's dependencies are beginning to drop support for Python 2, and time troubleshooting such installation issues takes away from our forward-looking work.

    3. Organise a bug triage/fixing day. We're planning to nominate a day where a group MediaGoblin contributors will make a concerted effort to resolve bugs. This is aided by having a team across many timezones.

    4. Automate testing of the installation process and test suite. Many of the questions we get to the mailing list are installation or dependency related. By automating our testing, hopefully across a number of popular operating systems, we should be able to reduce these issues and improve the installation experience.

    We'll look forward to telling you about our longer-term plans soon! For now though, from all of us hear at MediaGoblin, please take care of yourselves, your families and communities through the ongoing COVID-19 health crisis.

    Contributors in the last few years:

    • Andrew Browning (aleph): bug fixes and improvements, merging contributions
    • Boris Bobrov: bug fixes and improvements, merging contributions
    • Chris Lemmer Webber: bug fixes and improvements, community organising, ActivityPub
    • Jessica Tallon: bug fixes and improvements, ActivityPub
    • Vijeth Aradhya: multiple-resolition video transcoding/switching
    • Simon Fondrie-Teitler: upgraded/migrated/managed hosting for website, Trac, mailing lists and wiki
    • Saksham: lightbox, video subtitles
    • Amelia Rose: bug squashing parties
    • Ben Sturmfels: bug fixes and improvements, migrated mailing lists
    • Iyzsong: bug fixes
    • Berker Peksag: Python 3 bug fixes
    • Johnathan Sandoval: login validation improvement
    • Leah Velleman: localisation
    • Loic Dachary: cleanups
    • Matt Dealsy: UI improments (PyCon AU 2016 sprint)
    • Kesra: bug fixes (PyConAU 2016 sprint)
    • Josh Crompton: UI improvements (PyCon AU 2016 sprint)
    • Olivier Mehani (shtrom): OSM tiles, datetime format config
    • Alexandre Frankie: docs improvements
    • Stéphane Péchard: collection option to addmedia
    • Robert Smith: bug fixes
    • Simen Heggestøyl: bug fixes and improvements
    • Romain Porte: tags in API
    • Chrysn: EXIF rotation
    • DPG: license option updates
    • Dod: bug fixes
    • Michael Lynch: dev setup improvements
    • Ian Kelling: set up and imported our mailing lists (twice!)
    • ... and apologies to anyone else I've missed

    ActivityPub and MediaGoblin at TPAC 2016 (or: ActivityPub needs your review!)

    Screenshot of ActivityPub Working Draft

    Hello everyone! We have a lot of news to cover, but I'm going to jump right into the thick of it: we've been working hard on a new federation (as well as client to server) standard called ActivityPub (formerly ActivityPump). We've made tremendous progress, and I was just recently at a face to face meeting at TPAC, the W3C's big technical conference.

    The good news: ActivityPub is aiming to hit Candidate Recommendation status by October 11th. (That's less than a week away!) However, in order to enter that stage, we need your review! If you have any interest in the decentralized web, you can help. All you have to do is read the latest editor's draft and provide feedback. (The earlier the better... maybe a fun weekend project?) You can do this by any of the following:

    • Post issues on the issue tracker (yes, I think it's ironic and problematic that we're using GitHub, I picked my battles here, whether right or wrong).
    • Email the social working group comments mailing list.
    • You can email me directly. Email cwebber AT dustycloud DOT org, and include "ActivityPub" in the subject. Note that I will be publishing your comments publicly, probably on the Social Working Group wiki!

    If you have feedback, we want to hear it! (From anyone, but especially from those who are interested in implementing ActivityPub!) Thank you!

    ActivityPub live

    Note, the rest of this post is a bit of a dive into behind the scenes decisions and activity in MediaGoblin and ActivityPub... there's no need to read this part to submit a review!

    I'm kind of traveling through time in all the wrong order in this post... but it's worth jumping forward a bit to see the results of what we've done. Over the last many months there's been a huge push on ActivityPub as a standard, as we'll talk about. But in order to succeed, I also needed implementations, both of client to server and server to server. Well, I'm happy to say that those did come together...

    Pubstrate in action!

    This is Pubstrate, an implementation of ActivityStreams and ActivityPub for GNU Guile. Sorry for the sappy embedded love-note, though what's nice about this is that it's the first demo I gave to someone else of the client to server functionality working in Pubstrate. But wait, what was the client?

    Soci-El in action!

    This is soci-el, an ActivityPub client written in... you guessed it... emacs lisp. You can see the rendering of the user's outbox here as well as the buffers from which the message was composed.

    Pretty nerdy! I don't expect everyone to be using emacs as an ActivityPub client of course... I hope to see various desktop, web, and mobile clients made available. But, Emacs is the most fun for me, and I was time pressed, so there we are.

    Everyone loves seeing screenshots, but maybe not all of this stuff makes sense without context. What does this mean for MediaGoblin?

    So what's been going on?

    It seems a recurring meme in MediaGoblin land to say "we've been quiet, because we've been busy" (or maybe historically on every tech blog ever), but I guess I can't resist repeating the mantra. It's true! Though the weight of my focus has been shifted from where I expected it to be. From the last few updates over the last year, you would be right to anticipate that the main thing I would be working on would be merging the federation code Jessica has written and getting 1.0 out the door. That was the plan, and we're still working towards that, but priorities shifted as we realized the opportunities and time pressures we were under with ActivityPub. After the Social Working Group face to face meeting in June, Jessica and I sat down and talked about what we should do. Jessica had recently started working at Igalia (great people!) and was busy with that and other transitions in her life, so we discussed whether we thought it was most sensible to focus my energy on MediaGoblin or on ActivityPub. It was clear that ActivityPub was shaping into a solid specification, but it was also made clear that the Social Working Group's charter was running out by the end of 2016. We both think ActivityPub is key to MediaGoblin's success and didn't want to see our invested time go to waste, so decided my immediate focus should switch to ActivityPub so it could successfully make it as a standard.

    Which isn't doom and gloom for MediaGoblin! MediaGoblin development has continued... the community is good enough that people have been able to work while I've been busy. I'm happy to say we also appointed longtime contributor Boris Bobrov as co-maintainer to help reduce me being a bottleneck. (Thank you Boris, and congrats!) Other contributors have also stepped up to the plate. I'm especially thankful of Ben Sturmfels for hosting MediaGoblin hackathons and being so responsive to community members. (And of course, there are many others to thank, too!)

    Anyway, I'm not going anywhere, I've just been shifting focus to standards work for a bit... but it's all for the greater good of MediaGoblin. (Well, and the greater federated social web!) Soon, we'll be putting the work we're doing on ActivityPub directly into MediaGoblin. When we merge Jessica's work on federation, we will also retool it so that the first federated release of MediaGoblin will be blazing the trails with ActivityPub.

    Both ActivityPub and I personally got a significant boost by a happy visit from friend and Social Working Group co-conspirator Amy Guy. Amy dropped by for an intense four days of standards hacking and scheming, and came up with several significant ways to restructure and improve the document. With her help, we now have much clearer distinction between the use of ActivityPub as a client to server protocol (think mobile applications and desktop applications connecting to your server) vs the server to server protocol (federation). Both of these are now clearly intertwined in the document, but are distinct enough where they can be used and understood separately if desired. We also more clearly established the connections between ActivityPub and the linked data community by explaining ActivityPub's relationship with Linked Data Notifications.

    Amy has a gift for composing standards language, something I'm still struggling to learn (but hopefully getting better with over time). ActivityPub is much better with her hard work. Thank you Amy!

    In addition to the standards side of things, in order to get ActivityPub to the next level, we needed to be able to present real live demonstrations of the standard in action. Hence the work on Pubstrate and soci-el mentioned previously. For most of the months before TPAC, I was working furiously day and night to get things ready to show... And then, it was time to head off, and hope it was good enough...

    TPAC

    Live demo image, by aaronpk
    TPAC demo room photo by Aaron Parecki

    Fortunately, all that hard work paid off. The Social Working Group kicked off TPAC with live open-viewing demonstrations of the various standards we've been working on. We got in a really solid set of demos from everyone in the group.

    From my end, I managed to demo all the core parts of the ActivityPub spec: I wrote a note in my client (soci-el), fired it off to the server (Pubstrate), where it rendered successfully. Then I explained, well, what if you want someone on another machine to see it? So I had another user on a separate Pubstrate instance, added the user over there to the recipients list on my message, shot the message over to the server, crossed my fingers and hey! The live demo of federation succeeded. (Whew!)

    In the actual main Social Working Group meeting, we hammered things forward pretty nicely. As said, ActivityPub was positioned to move ahead towards Candidate Recommendation by the 11th. Again, your feedback is most valuable at this time!

    Spying image from the campaign video

    Oh, one more thing. On the second day of the main Social Working Group meetings, at Amy's suggestion (apparently she was impressed when I showed her at her visit), I showed off the MediaGoblin campaign video to the group. Apparently I had never done so previously, so it was really nice to hear the reaction: "Holy cow, this is describing exactly the type of stuff we're working on in the working group!" Yep, exactly... all that stuff you see illustrated in that video, we're working on putting into code and standards. The dream lives!

    So, all this Social Working Group stuff... things are happening! We're not just goofing off!


    Super serious TPAC group "photo" by Aaron Parecki

    ... er, right. Not just goofing off! :)


    MediaGoblin 0.9.0: The Three Goblineers

    MediaGoblin 0.9.0: The Three Goblineers
    "The Three Goblineers" by Christopher Allan Webber (pen and ink) and Morgan Lemmer-Webber (colored pencils). Licensed under CC BY-SA 4.0.

    This release is called The Three Goblineers, because we are finally fully embracing Python 3! You could even think of this release as Py-oneering, which it definitely is. Many traditional web service tools are less-than-ideal for federation and so we've had to do a lot of rebuilding and retooling. This release represents lots of intense behind the scenes work to make the user experience smoother, as well as some key improvements for MediaGoblin developers and deployers.

    Federated services use databases in a some fundamentally different ways. We had to make a traditional (rigid) database more friendly for more flexible relationships. The result is similar to the generic foreign key implementation used by Django, but optimized especially for federation. Jessica Tallon did the lion's share of this work and was aided by Andrew Browning who did extensive testing.

    We also updated how we handle comments and media collections. On a multi-media service, people will naturally want to reply to comments with videos and to ASCII art with songs so we had to make our commenting function support all the available media types. Also, the media collections aren't just for your personal gallery anymore; they're also used now in federation and the API as the backbone of a user's "inbox" and "outbox" feeds. Also, to make life easier on those uploading whole albums of content, if your user has some collections available, these will be presented as a dropdown option while submitting media.

    Selecting a collection during submit

    In the less visible but equally important department, we updated OAuth and created a "graveyard" system. We updated our code to make better and more secure use of OAuth so that my server and your server can be really sure we are actually talking to each other. The "graveyard" system uses tombstone icons to let you know that an object was removed by it's original poster. This makes it easy for users to remove media they no longer want to share, while creating as little confusion as possible for other users.

                  *                      *
     *                      *                  _.     *
          *           *          *            <  '.
                 *                             )  )       *
                                        *     <_.'   *
        *      *        .-------------.
                      .'               '.                *
           *          |                 |   *
                      |   TOMB OF THE   |       *
                *     |     UNKNOWN     |            *
       *              | ACTIVITYSTREAMS |
                      |     OBJECT      |
                 .^.  |                 |
      _  .+.  _  |~|  |    ????-????    |  .+. .-.  _  .-.
     | | |~| |=| | |  |                 |  |=| |~| | | |"|
    ``'`'`''``'`'`'`'``'``'`'`''``'`'`'`'``'`''``''``'`'`''
    

    Developers will be happy to know that we are keeping pace with the larger Python community and now have complete support for Python 3. We had early and experimental Python 3 support in 0.8.0 but couldn't quite recommend it for production use. That work is finished! Now you can choose to hack in Python 3 or Python 2 and your code will be totally compatible with the main codebase. Since Python 2 won't be supported forever, Python 3 is the future. And we are all about the future.

    We also switched our migration system over to Alembic. Previously, we'd been using sqlalchemy-migrate, but it also wasn't as good for updating the database layout when you upgrade, and lots of dragons be there. We think Alembic will make it less terrifying to update your MediaGoblin instance so you can stay current and without spending lots of time trying to sort what happened in the database. Alembic is also newer, fresher and better maintained!

    Another thing that makes updating a little persnickety is finding out that the dependencies have changed. We are collaborating and experimenting with the Guix community to fix this issue. Cleaner packaging will help people upgrade without fear, using deployments they can trust. Watch this space for a future post on how to use Guix and MediaGoblin together for worry-free upgrades.

    For users upgrading from previous versions, as always, check the release notes for instructions on upgrading!

    Our work to overthrow the red eminence of the centralized web continues! Between Python 3 support and laying down foundational changes to support federation, this release brings us much, much closer to our long term goals! Thanks to all our Goblineers and Py-oneers without whom this work would not be possible! Give a round of applause to Andrew Browning, Ben Sturmfels, Berker Peksag, Boris Bobrov, Christopher Allan Webber, Daniel Krol, Deb Nicholson, Duncan, jerome, Jessica Tallon, Loic Dachary, Sebastian Spaeth, Tom Fay, and 宋文武! You all helped make it happen!

    And if you want to make it happen in our next exciting release, we'd love to have you get involved! Visit us in IRC; #mediagoblin on freenode.net! Or sign up for regular updates on our mailing list. Got ideas or questions about our work? Email us at press@mediagoblin.org -- we look forward to hearing from you!


    Apply for GSoC in MediaGoblin (and Guix/Shepherd!)

    Hello all!

    Summertime is fast approaching, and this means GSoC is fast approaching too. This year we have some interesting potential projects. Check it out, and if you're interested, apply! You have until Friday (March 25th) to get your application in.

    We just added a new item, and I wanted to call it out specifically: porting federation to ActivityPub. This is not the only potentially interesting GSoC project, but it is a special one; you'd be able to work with Jessica Tallon (the source of federation support in MediaGoblin!) on updating federation to our current standards work.

    One more call-out: this one isn't MediaGoblin specific, but I (Chris Webber) am willing to mentor it: Adding an extensible event loop to Shepherd (the init system used by Guix). This would quite probably use the under-announced GNU 8sync project. If you're interested in learning about event loops and how to write them, you might want to watch and even better, read the corresponding sections of SICP which inspire it. This will be a tough one! But if you're really interested in digging into some cool ideas about concurrency, I'd recommend it. There are some other great projects for Guix as well!

    Both MediaGoblin and Guix welcome all applicants, though both projects also strongly encourage women, non-binary gendered individuals, people of color, and other underrepresented groups to apply. Both projects follow a Code of Conduct (for MediaGoblin and for Guix).

    Time is running short; apply! Detailed proposals are encouraged, and jumping in and experimenting / will greatly enhance your possibility of acceptance in both projects. Join #mediagoblin and #guix respectively on irc.freenode.net to talk to other developers.

    Happy hacking!


    MediaGoblin 0.8.1: Security release

    Basic Summary

    We have had a security problem in our OAuth implementation reported to us privately and have taken steps to address it. The security problem affects all versions of GNU MediaGoblin since 0.5.0. I have created a patch for this and released a minor version 0.8.1 (see the release notes page). It's strongly advised that everyone upgrade as soon as they can.

    In order to exploit the security issue, an attacker must have had access to a logged in session to your GNU MediaGoblin account. If you have kept your username and password secret, logging in only over HTTPS and you've not left yourself logged in on publicly accessible computers, you should be safe. However it's still advised all users take the following precautions, listed below.

    Users should check their authorized clients. Any client which looks unfamiliar to you, you should deauthorize. To check this:

    1. Log in to the GNU MediaGoblin instance
    2. Click the drop down arrow in the upper right
    3. Click "Change account settings"
    4. At the bottom click the "Deauthorize applications" link

    If you are unsure of any of these, click "Deauthorize".

    I would like to thank Dylan Jeffers (author of Goblinoid) for finding and reporting this to us in a responsible manner so that we were able patch this.

    Technical Information

    The security issue was caused by the verification of the OAuth verifier code. There the proper checks were not occurring to validate the verifier code matched the one issued to the request.

    This only affected those who clicked the verifier link whilst being logged in and entered a different code. The assignment of the user to the access token only occurs when you go to the authorization page whilst being logged in. If the link isn't clicked with the user logged in no user will be assigned to the access token and a client attempts to use it will be denied as the endpoints won't be able to look up the requesting user.

    A patch has been made should you wish to view the fix.


    State of the Goblin: Stripe Open Source Retreat, and more!

    Hello, all!

    It's been a few months since my last major update so I wanted to fill in what's going on. As usual, a lot has been happening, and it's been hard to cover it all as we go. There's some particularly huge news in this update, including something about funding something oh hey this should help us get MediaGoblin 1.0 out the door, plus something about the standards work we're doing, something something. So let's dive in and resolve all those somethings, right?

    Support organizations that support freedom!

    Okay, wait, a brief intermission! We'll get to the cool MediaGoblin related news in a moment, but we've got something very important to cover first. Two organizations I really care about are running funding campaigns. Okay, well, it's that time of year, and a lot of organizations are running funding campaigns, but these two especially could use your kind contributions!

    Support Conservancy!

    Copyheart!
    Copyheart by Christopher Allan Webber, CC BY 4.0 or GPLv3 or later, your option.. Source here.
    Become a Conservancy supporter!

    The first is Software Freedom Conservancy. They're close friends of ours, and do great work. They're also pushing hard to try to build up a supporter program, and they could really use your help. Conservancy does a lot of work around many things, from running Outreachy (hey, we were lucky to get Jessica Tallon working on our stuff through that!) to enforcing the GPL to hosting a whole lot of useful free software projects. Become a supporter today!

    Support the FSF!

    Next up is the Free Software Foundation. Hey you probably know these folks right? The FSF is the steward of GNU, and MediaGoblin is a GNU project, so success for the FSF is success for MediaGoblin. They're running their annual fundraising campaign and they could really use your donation. The FSF has the long-standing history of being the anchor of the free software world, and they continue to do great work year after year. Help the FSF continue far into the future... get your donation in today!

    Support Guix!

    GuixSD
    GuixSD logo by Luis Felipe López Acevedo. Permission is granted to copy, distribute and/or modify this work under CC BY 4.0.
    Support Guix!

    Did I say two organizations? What's with three sections? In fact, our friends at the FSF are also teaming up with our friends in the GNU Guix project to run a special campaign to raise funds for new servers. I'm highlighting this for two reasons:

    1. I believe Guix is very important to the long run of "deployability" in free software network services (which you may know I believe to be an extremely important issue...)
    2. Guix is using a similar funding model of going through the FSF as fiscal sponsor. This is the same route we took for the MediaGoblin campaigns and I think is a great way for free software projects, and particularly GNU projects, to go. I'd love to see further examples of success for it in Guix.
    So, go donate! :)

    Okay, whew... sorry about that distraction, but these things are really important! But I know, I know, you came here for the MediaGoblin news. Well, let's get to that!

    MediaGoblin selected for the Stripe Open Source Retreat

    So this is some big news! MediaGoblin has been selected to be a participant in Stripe's Open Source Retreat 2016!

    What does this mean? It means that I'll be moving to San Francisco from mid-January to mid-April 2016, working from Stripe's office, and Stripe is going to pay me to focus totally on getting MediaGoblin 1.0 out the door and advancing our federation work. This is a huge opportunity for us; getting such ~unrestricted funding is, as anyone who has ever done fundraising knows, enormously difficult. We should be able to use this to bring MediaGoblin to the next level.

    When I filled out the application for this I was interested but skeptical of Stripe's claim that this would be "no strings attached" funding. I'm happy to say this seems to be true: they've paid us to do the work, and Stripe is making no claim to copyright or asking us to change any of our existing policies. Contributing upstream to MediaGoblin happens as usual for the time I'm there, which is great.

    So what do we hope to get out of it? Well, my goal is by the end of the retreat, we'll have MediaGoblin 1.0 out the door with the basics of server to server federation in place. I've also talked with the Stripe team about using that time to advance work on our federation standards work, and if there's time, some deployment work too. But MediaGoblin 1.0 comes first!

    (In the meanwhile I just booked a bunk bed for $1000 USD per month, which it turns out is cheap for San Francisco housing. Egads! How do people afford to live there? Luckily it's well covered by the retreat's stipend! Are you in the area during that time? Maybe we should meet up!)

    I'm very excited about this opportunity. Thanks again to Stripe for supporting our community. I promise that the grant will go to good use, and we'll have exciting things to report!

    W3C updates

    W3C Social WG, third GMG represented meeting
    W3C Social Working Group December 2015 face to face meeting attendees. I'm hiding in the back.
    Photo taken by Aaron Parecki, CC0 1.0, originally posted to the W3C wiki.

    Our work to standardize federation technology within the Social Working Group continues. Just a few weeks ago another face to face meeting was held at Mozilla's San Francisco offices (thus giving me an excuse to test-run the bunk bed I'll be sleeping in for several months for the Stripe retreat). The meeting was extremely productive... in fact I would say it was the most productive time the Social Working Group has ever had. You can read the minutes for Day 1 and Day 2 if you like that sort of thing. But here are some highlights:

    • Most of the work towards making ActivityStreams 2 a Candidate Recommendation document was put in place, and in a couple of weeks this I expect it will achieve that goal. Candidate Recommendation is a big step in a standards process, and AS 2.0 is the heart of ActivityPump, so this is huge for us.
    • ActivityPump, the standard we are pushing for server to server federation and client to server communication has moved to Editor's Draft state! The objective of moving to First Public Working Draft by mid-January has been set, and we are pushing hard towards it.
    • Because of increased push on moving ActivityPump to this state, I have been added along with Jessica Tallon as co-editor on the specification.
    • The "IndieWeb stack" (for lack of a better grouping) standards have all also advanced to Editors Draft status, including Webmention, Micropub, and Post Type Discovery. These standards are also on track for First Public Working Draft in mid-January.
    • Amy Guy's work on a "convergence" standard has been renamed to Social Web Protocols and also (are you noticing a trend?) advanced to Editors Draft and is on track to First Public Working Draft in mid-January.
    • I demoed ActiviPy, which went over really well. ActiviPy started out as a method of representing and working with ActivityStreams for Python, but I realized in developing it that since it was using JSON-JD with an implied context to handle ActivityStreams anyway, I could also extend it to support the Microformats vocabulary by using the JF2 context being developed for JF2. So I showed off a demo where I loaded ActivityStreams2 documents, demonstrated the method dispatch system ActiviPy uses (which is fairly interesting but I won't bore readers with it), but most excitingly, I loaded ActivityStreams and Microformats documents side by side in the same system and then converted them both to linked data! This got a fairly strong reaction from the room, since this was all three of the "directions" of achieving federation we've been working towards, with a real live demo of convergence! I was very proud to show this off.

    Maybe most importantly was the "spirit of the room", and how much this has changed from prior meetings. This group was formed to work on some very challenging domains with the goal of bringing initial participants with some historically very differing backgrounds. But both the last face to face (which we mentioned in the last state of the goblin post) and this one have really done tremendous things towards propelling this group forward, and unbelievably, towards something that might actually be convergence (without requiring that).

    Those who know enough historical detail of this space may be astounded to read that last sentence, but I believe it is true. In a sense, agreeing that convergence was not mandatory helped bring us towards a greater possibility of it. The agreement that ActivityStreams, Linked Data, and the "IndieWeb Stack" (for lack of a better term) were not required to work together, and that we could produce multiple deliverables, has eased that tension in the group and allowed us to work collaboratively. Everyone has worked hard to understand each other. But one person in particular has been doing a stand-up job of trying to bridge the cognitive gap and that person is Amy Guy. This can be seen immediately with her work on the Social Web Protocols document, but I believe she has done a great job otherwise in mapping the space.

    So anyway, optimism can only bring you so far. There's a ton of work to be done in this space, and we'll be pressing hard.

    See you at FOSDEM!

    Are you going to FOSDEM? So will I! I'll be giving two talks in the Guile/Guix room, and I may be giving one more in another room, depending on acceptance or not. It would be good to meet other MediaGoblin community members or enthusiasts of MediaGoblin.

    I should also say that I was really not sure if I could make FOSDEM originally, but a number of people very kindly donated to send me on my way. Thanks very kindly! I'll be sure to put it to maximum use for our community's sake. :)

    We're also thinking of running some sort of dinner (or lunch?) for those who donate between now and FOSDEM to MediaGoblin, so hey, by the way, our donate page still works! ;)

    Goblinoid updates

    Goblinoid, you may remember, is the result of this year's Google Summer of Code, and is a neat MediaGoblin application for Android. Here's an update (thank you Laura Arjona and Dylan Jeffers for writing this up):

    Dylan Jeffers, our GSoC 2015 student, is working in an Android app for GNU MediaGoblin.

    Current features include viewing the recent activity feed, comment about media, upload photos from file... taking advantage of the Pump API.

    The code is under heavy development, (repo here), and binaries for each release can be found (along with checksums) hosted at Goblin Refuge, a third party site kindly offered and maintained by SalmonLabs LLC, who also host the goblinrefuge.com MediaGoblin public instance.

    Meanwhile we get the app in the popular free software repository F-Droid (it's taking a while because it's the first app in F-Droid built using the Python-based Kivy framework), we encourage everyone to test the app. The most up to date release is always available at https://files.goblinrefuge.com/download/Goblinoid/MediaGoblin-latest.apk) and report feedback in our IRC channel or the repository issue tracker).

    We are very excited to make MediaGoblin part of your mobile life, and the Android app development is allowing improvements in MediaGoblin itself too (mainly fixes/improvements in the API-related code and the database and database migrations). Help us with your testing to improve that experience!

    Wrapping up this year, onto next!

    It's been a busy year. Here are some highlights:

    • We took initiative on the challenges of user-centric hosting.
    • We got out a new release of MediaGoblin and dug in for the hard work ahead of getting towards 1.0.
    • We put your money to good use and funded Jessica to plow ahead with federation in MediaGoblin. This is well on its way, and we anticipate server to server federation to land in the next couple of months. Jessica just landed a massive overhaul to our database structure which was required to make this work, and that should make its way into the next release (coming soon).
    • We became more engaged with the work to bring federation beyond MediaGoblin itself. Jessica and I both joined the W3C Social Working Group and have become co-editors on the ActivityPump specification and have devoted much time to advance these initiatives. This includes building tooling such as ActiviPy which will be critical for putting federation to real use within and outside of MediaGoblin.
    • For the second year in a row, a major MediaGoblin member has received the O'Reilly Open Source Award largely (though not only)in recognition for their work on MediaGoblin. This year I (Chris Webber) received the award and last year Deb Nicholson received the award, which is rather incredible given the list of previous recipients of the award.
    • Both Deb Nicholson and I (but especially Deb) have given talks at prominent conferences and have spoken on podcasts on issues of network freedom.
    • We had a successful Summer of Code project resulting in Goblinoid, a free software client for MediaGoblin, which is Android and Replicant compatible. (It can run on GNU/Linux also!)
    • And as we said, we were accepted as recipients for the Stripe Open Source Retreat! (Well, accepted this year, and the results of that should play out into 2016!)

    That's a lot of stuff... what a year!

    Well that's it from this update! See you in 2016, and happy hacking, fellow goblins!


    State of the Goblin: September 2015

    Quick announcement: I'm going to be making two appearances over the next week! First I'll be giving two talks on September 30th at Red Hat's Chicago office for the Chicago GNU/Linux User Group on Guix and Federation (both mentioned in this post)! Second I'll be attending the FSF 30th Birthday Party in Boston on October 3rd. If you're able to make it to either, do stop by and say hello... I'm expecting both to be a lot of fun!

    Hello everyone! It's been a while since a comprehensive update of what's happening in MediaGoblin land. Despite the quiet, there is a lot to report, so let's get down to business and start reporting!

    O'Reilly Award (again!)

    receiving the award
    Photo taken by Brandin Grams, CC BY 4.0, originally microblogged by Karen Sandler

    First of all, something fun: I was fortunate enough to receive the O'Reilly Open Source Award! (Yes, I know about the terminology mismatch, we're free software people, but it's still a great honor, and I was presented the award under the description of my free software advocacy and GNU MediaGoblin work.) The other recipients of the award is quite the incredible group of people, and I'm honored to be listed among them. But here's what's really cool: you may remember that MediaGoblin co-founder Deb Nicholson won the O'Reilly Award last year. How's that as a vote of confidence in the things we're working on?

    O'Reilly award, on display

    Anyway, if you want a more personal reflection, I wrote more on my personal blog!

    Releases

    MediaGoblin 0.7.0: Time Traveler's Delight banner

    So right, what about shipping software out the door? Well...

    MediaGoblin 0.8.0: A Gallery of Fine Creatures banner

    Since the crowdfunding campaign, we've gotten out two major releases, 0.7.0: Time Traveler's Delight and 0.8.0: A Gallery of Fine Creatures. I'm extremely proud of both of these releases! We have a lot more to do though on the road to 1.0, and we've been directly been putting the funds from the campaign to work to achieve that goal, so let's talk about that.

    Putting your money to good work: Jessica and Federation

    Dropdown menu for administrative features

    You may recall that we hired on the talented Jessica Tallon to get federation working in MediaGoblin. Jessica recently gave an update on the state of federation. Jessica is doing great work, though as expected, converting MediaGoblin to be a federated project has been no small task (knowing what a big task it was, hope that we could hire Jessica on to do this work was my #1 goal in the last campaign, in fact!). This decision has turned out to be absolutely the right one. Some of the best parts of the last two releases have been adopting the client to server Pump API. Federation has been MediaGoblin's goal since day 0, and Jessica is helping us to actually get there.

    However (and now I'm going to do a pretty technical deep dive, so you can skip this paragraph if that isn't your thing), the most complicated aspect to making MediaGoblin into a federated project has had to do with updating the database to handle things while preserving data correctly for existing users. Why is this so complicated? A number of years ago we switched MediaGoblin over from using MongoDB to using either PostgreSQL or SQLite and while I believe this was absolutely the right decision, adding federation made the relational database system we had in place substantially trickier. For the more database-technically inclined, you can see that the ActivityPump API / Pump API require that any ActivityStream type object (in our case, that can be media or comments or even users) be referenceable by any type of activity. Furthermore, our existing comment system simply held that comments referenced media entries, whereas now comments can reference simply anything that is an ActivityStreams object. This means a large portion of our relations in our relational database needed an overhaul, and we needed a way to handle generic relations between tables. (The solution used is not unlike the "generic foreign key" implementation in Django.) There are more technical details on what has been done, but Jessica has been neck deep in this for months, but we believe we're finally on the home stretch, in which case Jessica can finally knock out server to server federation.

    (I've thought that a whole post on database structure lessons learned may be a good blogpost of its own. One thing I'd note is that if jsonb had been an option when our current database design was put together, adopting that would have simplified things greatly, though it would require being PostgreSQL-only. But moving to that now would require a massive overhaul. If you're starting a new federated project from day 1, maybe keep that in mind!)

    So the summary of all the federation stuff is: it's complex, but we're making good progress through Jessica's hard work. Expect more on this soon, and huge strides in the next release!

    Federation and the W3C Social Working Group

    So something Jessica and I have both been involved in over the last year is the W3C Social Working Group working towards official standards for federated web applications.

    W3C Social WG, first GMG represented meeting
    W3C Social Working Group March 2015 face to face meeting attendees. Jessica's holding the laptop on the left, and I'm right behind her.
    Photo taken by Aaron Parecki, CC0 1.0, originally posted to the W3C wiki.

    The federation protocol that MediaGoblin has been working towards until this point is primarily based on the Pump API, but this is really just a semi-formalization of the interface for the pump.io API. In the Social Working Group we are working towards defining a new standard, ActivityPump, which is based off of the ActivityStreams 2.0 standard. We're very excited with where this standard is going and feel it's a clean refinement over the Pump API we're already working with, while still keeping many of those same conventions.

    W3C Social WG, second GMG represented meeting
    W3C Social Working Group May 2015 face to face meeting attendees. I didn't make it to this one, but Jessica did, and did a stellar job representing MediaGoblin and ActivityPump!
    Photo taken by Aaron Parecki, CC0 1.0, originally posted to the W3C wiki.

    This has taken a lot of our time, but I believe it the results are worth it. Jessica and I have been attending weekly calls related to this standardization, and have thus far attended two face to face meetings at well. (More accurately, Jessica attended the second MediaGoblin-represented one without me, giving a kick-ass presentation on how ActivityPump works to the group! Go Jessica!)

    As for my own personal work advancing this, I'll go into this a bit further on in this post!

    Google Summer of Code result: Goblinoid!

    Current screenshot of Goblinoid
    Screenshot of Goblinoid, as it looks now!

    Dylan Jeffers joined us for Google Summer of Code student this year work on a pretty cool project: a MediaGoblin client for Android or... really nearly anything... called Goblinoid! There's two really interesting features about Goblinoid: one, it's written in Kivy, a GUI toolkit which emulates the Android look and feel, but is actually can run nearly anywhere Python can run... making it quite portable, yet ideal for mobile computers!

    Mockup of Goblinoid future UI
    Mockup of what Dylan would like Goblinoid to look like in the future!

    So Goblinoid works... it could use more user testing and packaging, if you're interested in helping with that! But you can already upload images on the go via Goblinoid, and we expect more to come. Give it a go!

    We've long been interested in having a client for MediaGoblin which makes use of the Pump API implementation we've been working on. Thank you Dylan for making that happen!

    Infrastructure challenges

    This has been a challenging year as in terms of supporting MediaGoblin's infrastructure. Spammers attacked both our wiki and bugtracker hard, at one point leaving me to take several weeks to fight issues and to try to find solutions. (Unfortunately, for the bugtracker, no great solutions have been found, and we are on a request-an-account basis... not a great situation to be in.)

    Additionally, the primary Gitorious instance went down, where MediaGoblin's code was hosted along with many, many feature branches from contributors. The MediaGoblin community spent a while debating what we were going to do. A move to GitHub was tempting but is not an option; that's exactly the opposite of the type of world we want to build. Not everyone in MediaGoblin's community is comfortable with GitLab, and their primary instance is running a proprietary version. There are some other communities hosting things like Notabug who seem to be run by great people, but we run the risk of running into these same problems all over again... though so we did with most of these other solutions. We could self-host, but there is very little time for extra server maintenance burdens right now. So we moved our git repository over to Savannah and I'm glad to have hosting there by people I trust, though I do also feel that contributors may expect more modern hosting facilities, but we don't have time to run them ourselves.

    The Gitorious shutdown came around the time of great exhaustion of already dealing with issues related to the bugtracker and left me feeling very burnt out. But it also lead to a great amount of reflection... who am I to feel frustrated with? The Gitorious people graciously hosted our software repositories for some time, and I was not willing to run an instance of my own. Why not? Well, one problem is that if you run your own instance of a free software web application that isn't federated, you're working in Yet Another Semi-Free Micro-Silo (TM). But let's face it, that's not the real major problem... looking at the frustrations we've had with Trac, the answer is obvious: running free network services is a huge maintenance burden.

    But wait... yes, you may be catching a whiff of irony here... if I'm not willing to run a free software web application because I don't want to take on the maintenance burden of someone else's software, how can I ever expect MediaGoblin to gain adoption?

    And here's where I come to a tough, but I think necessary, conclusion: there's simply no way for MediaGoblin to succeed if the world of deployment stays where it's at. Something must be done. But what?

    Research into deployment and federation

    Partly affected by the above, summer came around, and I had a talk with MediaGoblin contributors in our monthly meetings. I wanted to take a sabbatical... a sabbatical where I was on break from "direct" MediaGoblin things, so I can advance things that affect MediaGoblin greatly indirectly. (My spouse and I were also going through a big move so this was a good time to do it, I figured.) I wanted to do research into two things: deployment and federation.

    Framing the deployment side of things, Deb Nicholson and I gave a talk that kicked off "userops", a term that I still feel accurately captures what we're trying to do (and a talk which I still believe accurately summarizes the challenges we're facing). Within this context I began exploring options of what can be done to improve deployability.

    The directions I've explored, and why I've come to the conclusions I have, are a series of blogposts of their own (if you're interested, I suggest subscribing to the userops mailing list, where I will be posting more as I go). The short of it is that I laid out a set of requirements to achieve easy and maintainable deployments, attempted to explore them with the most popular current tools (Ansible, Puppet, Salt, Docker, etc), and found that it is not possible to build both easy and maintainable systems on top of them based on what I believe users need. This lead me eventually down the path towards Guix, a package manager (and with GuixSD, also a distro) and soon to be deployment system which I am very confident has the potential to solve many of the challenges which make deploying and maintaining systems too exhausting a task for the average user. The software is nowhere near being "easy" at present (I think it's very telling to say that it's "the Emacs of package managers / distributions"... one can extrapolate on that in many ways, most of which are correct, except for the Emacs-haters kind... lay off, Emacs haters!) but I think has the potential to become so. An easy to use web user interface has already been demonstrated, and I believe the foundations are good for building a complete and easy to use system for everyday users. But again, to go into detail beyond what has already been said is something that will be explored elsewhere.

    There is a more pressing need for me to have explored deployment at present as well... though I want to explore deployability for the sake of other users, I also need to explore deployability for my own sake... particularly because we promised that we will provide premium hosting in the last campaign we ran! Yes, in case you have been wondering about it, I have not forgotten about this promise. How to fulfill this promise without being crushed by the maintenance burden of hosting? I came to realize that there was every risk that I would spend all my time supporting and maintaining servers that were running MediaGoblin and I would not have the opportunity to any longer be a steward of MediaGoblin itself... which could lead to failure. So figuring out a better path forward on hosting has become a necessity. I'll explore what this means further in the next section, but first, I should say a word on federation and what my sabbatical lead to on this front.

    As I have said, Jessica and I have been involved in the W3C Social Working Group. Part of the activities of the group have been the definition of the ActivityStreams 2.0 and Pump API standards.

    I set out to explore this a bit more deeply under a repository with the joking and interim name of activitystuff (the Social Working Group is using GitHub for its work, so I'm making an exception there). Along with some other projects, this has contributed to a deeper understanding of how federation should work, which is something useful to take back to MediaGoblin. There are some potentially useful things in that repository (including a partially complete implementation of the JSON-LD API), and it may turn into a full implementation of ActivityPump, though its original and primary goal was for exploration, a purpose it has served well.

    One major event relating to federation has just occurred over the last week and bears note here: a number of Pump.IO community members and myself are now working with Evan Prodromou (who has near-certainly contributed more to the federated web space than anyone else alive) to transition the project from being a project of primarily stewardship under Evan to one of community stewardship and governance. I posted a summary of a recent meeting, and (MediaGoblin contributor!) Laura Arjona put together a community document. In sum, Pump.IO could use your help if you're interested.

    So anyway, it's been a busy last couple of months! But it's time to return to MediaGoblin-ville, so...

    What's next?

    Well, that's a whole lot of text above, so how about a bulleted list next? I hear those are easier on the eyes.

    • I'm swooping back into MediaGoblin territory starting next month. However, my initial focus will be on getting MediaGoblin to a deployable point for myself, particularly forward-looking towards making premium hosting feasible. Expect more soon!
    • Jessica is working on MediaGoblin federation. We expect the massive database changes she has been working on to tidy up in the next few weeks, and if all is well, we'll have server to server federation basics in place by the end of the year.
    • Work towards 0.9.0 will proceed, unless 1.0 lands first! Expect major infrastructure improvements in 0.9.0, and federation to land in 1.0.

    Those are all great things, and I think we are indeed on track towards our goals, slowly but surely.

    There is a more rough side to this: we've allocated nearly all of the funds from the last campaign towards paying for Jessica's work on MediaGoblin, and she's devoted enough to the project that she's been working well, well below market rate. (And aside from some travel reimbursements, I have not taken money personally from the last campaign.) I'll post a financial transparency report soon, but the short of it is: the MediaGoblin funds are running low. Even with Jessica generously working for such a relatively low amount of money, we won't be able to pay Jessica to work on MediaGoblin much longer, given our present finances.

    But we aren't giving up! The goals of MediaGoblin and of federating the web are just too important, and so onward we press, into the future! Though MediaGoblin is an ambitious project, I am confident we can achieve the things we have set out to do, but we are constrained by limited resources and time.

    Appreciate what we are doing? Want to help us in our quest to bring network freedoms to everyone? You can help!

    <3Donate...

    • The simplest way to help? Donate! Though the official campaign is over, you can still donate to MediaGoblin through the FSF!
    • Jessica and I have been contracting so we can pay the bills (doing this allowed us to "stretch out" the amount of time Jessica could spend on the project... useful with all that W3C stuff in progress!). Unfortunately, while the people we have been working with are great, the contract we've been working on is coming to an end. Are you looking for contractors or part time workers who are capable of developing high quality free software and providing community leadership? Jessica and I are both interested in working in such cases... feel free to email me at: cwebber AT dustycloud DOT org

    We're committed to making a better, decentralized web. I hope this piece cleared up where we're at and where we're going. I believe we've got exciting times ahead... till next time, goblins!


Page 1 / 8 »