Fun with Federation: Lemmy edition
-
It all started with a report about federation breaking between Lemmy and NodeBB. I was subconsciously aware that something was going on, but had chalked it up to network issues.
Observed behaviour showed that some remote categories would be receiving content in spurts, with long gaps in between.
I spent the next 3-4 days looking into it, but came up empty. Whatever was happening wasn't throwing any obvious errors, and along the way, I found what I thought was related (it was), but I wasn't sure why: against some Lemmy servers, the "follow"/"unfollow" mechanic would simply stop working, and this would often coincide with gaps in content. In some egregious cases, the flow of content stopped completely!
Unable to make headway, I had to reach out to the folks at Lemmy to figure out what the issue was. NodeBB occasionally sends non-200 level responses depending on the activity. Specifically, the following scenarios:
- A remote user upvoting more than 20 posts in a single day (a spam prevention tactic) causing NodeBB to throw an error, which was caught and returned an
HTTP 500 Internal Server Error. - A
Dislikeactivity, which is not currently handled by NodeBB. In these cases, NodeBB would send anHTTP 501 Not Implemented
When encountering either of these responses, Lemmy would return the activity back to the queue for later delivery and mark a delivery failure. If enough of these (~40) happened within 24 hours, Lemmy would give the instance a time-out and pause delivery completely.
That was it — a quick pair of code updates later, and we started working through Lemmy's backlog of 4.1M activities.
As of 4am this morning,
community.nodebb.orgis no longer behindlemmy.world.
Fun week. Let's not do that again LOL.
- A remote user upvoting more than 20 posts in a single day (a spam prevention tactic) causing NodeBB to throw an error, which was caught and returned an
-
These little inter-platform quirks are fascinating, and I imagine quite frustrating.
-
This is really cool. Thanks for sharing.
-
kichae@wanderingadventure.party I'm still at the point where seeing two-way federation working for real between instances is so exciting that it overrides the pain of federation
-
nutomic@lemmy.ml let me know if I got any of the details wrong. Much thanks to your team for the assist in debugging!
