Kosmos Mission Control @ops@kosmos.social

Sorry for the hickups on kosmos.social and wiki.kosmos.org since the early morning. One of our load balancers was malfunctioning out of nowhere. Should be back on track now.

We are going to switch our database primary node to a new machine. Not expecting any downtime. If something breaks, please just try again soon. Thanks for your patience.

For our Tor users:

We have updated the onion address for kosmos.social to a new v3 address, and we have also changed the setup to use HTTPS now, which will trigger a warning in your browser. Address and info here:

https://kosmos.social/about/more#tor

We just migrated kosmos.social to a new server, and updated Mastodon to version 3.2.2 in the process.

One notable change is that, in case you don't have 2FA set up yet, you will now be asked to verify new logins via email.

We're migrating gitea.kosmos.org (away from Google's GKE, to our own servers) right now. There should be no downtime.

... all done and services back up.

We're taking kosmos.social as well as xmpp.kosmos.org down for maintenance in a few minutes. We'll be upgrading the database server in preparation for setting up database replication to secondary servers...

We just enabled audio/video calls for your Kosmos XMPP accounts. Available for example with the new version of Conversations for Android.

Sorry, everyone! We just recovered from an unexpected downtime. Both Mastodon and XMPP are back now.

Our main machine went down last night, for as of yet unknown reasons. The machine was unresponsive and reboots did not work. We were able to have a remote rescue console attached and have fixed the bootloader now. It does not look like we lost anything. We're investigating the issue now.

Upgrade is done. Check out the new announcement for details.

Heads-up: we're upgrading kosmos.social in a minute...

The ops team is now hanging out in a new XMPP chat room. Come talk to us in ops@kosmos.chat!

xmpp:ops@kosmos.chat?join

All done with the gitea.kosmos.org experiments. You can safely push your code again.

We're experimenting on gitea.kosmos.org deployments a bit today. Please expect some downtimes. Thank you.

kosmos.social was down for a short while just now, due to a manual system upgrade throwing it off the rails. Sorry about that!

kosmos.social just suffered a short downtime, due to a system software upgrade not agreeing with our configuration variables.

Details, in case you're interested: https://gitea.kosmos.org/kosmos/chef/pulls/108

We just upgraded kosmos.social to Mastodon version 3.0.1 (from 2.9).

Release notes for 3.0.0, which contains all the new features: https://github.com/tootsuite/mastodon/releases/tag/v3.0.0

And there's even a video that walks you through the notes this time: https://www.youtube.com/watch?v=aYHYxOeQumo

Hi everyone!

This account is taking over from @kosmos for status and incident updates, as well as general infrastructure talk.