[Box-Admins] Squeak Server state of the Union or Hold Your Horses but Not Too Long

Fri Oct 6 06:58:40 UTC 2017

Hi all,
Re GCE and AWS: I quickly used their calculators to get some rough
estimates and I got 116$/month for GCE [1] and 197$/month for AWS [2].
In general, GCE is cheaper than AWS. We would be able to scale the machines
up and down, so it's possible that we actually pay a lot less, or a lot
more. I also have no idea about our current data transfer, so that could
also add to the bill. But at this point, it's quite hard to tell how much
we would actually pay without knowing more about our current
setup/resources.
Having said that, both GCE and AWS are probably to "big to fail" compared
to a web hoster from around the corner and the prices are still going down
every year without having to migrate anything.

No matter what hosting solution we choose in the end, I'm happy to help
with the next migration.

Best,
Fabio

[1]
https://www.dropbox.com/s/47f46cw4rsm0eeb/Screenshot%202017-10-05%2022.50.49.png?dl=0
[2]
https://www.dropbox.com/s/28qyrdb6orusfmn/Screenshot%202017-10-05%2022.51.37.png?dl=0

On Thu, Oct 5, 2017 at 2:37 PM Tobias Pape <Das.Linux at gmx.de> wrote:

> Dear All
>
> As the primary person responsible for the current server setup and
> maintaining some contact with the sfc admins, I'd like to give a bit
> insight into the situation.
>
> I. The Past
> ===========
>
> Around this time last year we started migrating our services from Gandi to
> Rackspace.
>
> This was because (a) the Gandi service was not very responsive and (b)
> about to be shut down December 2016. But also (c) our server setup had been
> "worn out" over time[1] and (d) the SFC had for a long time offered to host
> the services on Rackspace.
>
> Seeing that time was about to run out, I stepped forward and, admittedly,
> relatively singlehandedly decided the new setup. Having the disk-shortage,
> interference between services and the effect of 'dragging down' other
> services when dying in mind, I decides to "spread out" all services as it
> seemed justifiable. This was mainly due to the _very_ generous offer that
> we were allowed to use Rackspace services up to an amount of about $2000.
>
> Therefore, I created a little network of Squeak-related services. I had
> separated the services into web/files, mail, squeaksource (
> source.squeak.org and squeaksource.com separate), mantis, others (wiki,
> map), and later planet.squeak and codespeed. Every server is backed up
> individually. We make also use of one Rackspace MySql database.
>  The ideas here were:
>  1. Have only ONE ssh entrypoint to the squeak services (this is why we
> have the ssh gateway)
>  2. Isolate the services as good as possible
>  3. Avoid being offline when (a) a server crashes or (b) someone breaks in
> or (c) some admin makes a mistake.
>  4. Have backups of the services.
>  5. Have headroom on each server to be future-proof.
>
> Additionally, I had hoped that this service would continue for a long
> time, so that only minor adjustments would be necessary by the admins.
> Also, I wanted to minimize attack surface with this setup, so that one
> compromised service wouldn't expose others. The logs suggest that this
> threat is not theoretical.
>
> All that contributed to me setting up 8 servers with 1..2GB Memory, and
> 20..300GB HDD and 1 or 2 CPUs each (plus a Rackspace managed MySQL DB).
> They are all backed up by Rackspace.
>
> And, oh, our DNS is managed there, too.
>
>
> Since September 2016, we hat one infrastructure outage, which was the
> database for one hour in the middle of the night.
>
> So far so good.
>
> II. The Present
> ===============
>
> The setup as we use it would cost around $500–$600 per month if we had to
> pay it ourselves[2].
>
> But now we most probably have to leave[3].
>
> To be clear: I _do not_ suggest that we continue in the very setup we have
> and try to replicate it elsewhere.
>
> However, I think it is a good idea to _know_ what we are actually using. I
> tallied all services and servers and so on[4] and came to the conclusion,
> that we currently use the equivalent of
>
>  - 2-2.5GB RAM
>  - 200 GB Disk
>  - 4-6 CPUs
>
> [NOTE: THESE ARE NOT EQUIVALENT TO REQUIREMENTS FOR A NEW SETUP!]
>
> These services should therefore be satisfiable with a setup <$100 month.
>
> III. The Future
> ===============
>
> I think that it is perfectly possible to have a setup with 2 (Two) or 3
> (Three) servers.
>
> I would not suggest to consolidate everything, at least web/files and mail
> should be separated. But if there's no room for that, yes, we would be able
> to have everything on one server.
>
> That being said here are a few things to consider for the Future Squeak
> Infrastructure:
>
> * We need at least a month to migrate.
>
> That means, we must start end of the month THE LATEST.
>
> * We need a new DNS provider fast.
>
> I strongly advise against running a DNS server ourselves again[5].
> Also, Networksolutions has a very poor user experience.
> I personally use INWX[6] and it works well but any other decent registrar
> should work.
> However, this is to be coordinated with Dan Ingalls and Goran Krampe, as
> Dan owns squeak.org.
> Similar stuff to be done for squeakfoundation.org. (and maybe squeakland
> because of mails).
>
> * We need a new hosting provider.
>
> We have two or three main ways: Pay for ourselves, ask people nicely,
> (and: hope the SFC figures something out in time, but I wont bet on that)
>
>   - Paying for ourselves could mean several things, like renting servers,
> webspace, or the like.
>     We would probably try to reduce our service footprint even further,
> by, eg. hosting part of the website with github.com (via pages or the
> like) or moving from Mantis to github issues, trello, frogbuz or so, but I
> don't think that would do much in terms of footprint.
>
>     NOTE: I do estimates with 2 servers w/ 2-4 CPUs, 8 or 16 GB, >200GB
> HDD each.====
>
>     There are several options, but I personally only know a few
> (Euro-centric, because, you know...()
>     # hetzner.de has been our host for some time and we would probably
> come around some 40-100€/month
>     # netcup.de is what I use personally and they are a tad cheaper
>     Those have the problem that we would have to care for backup manually
> (they both offer space for that, at a fee, but without automatization)
>     I think there are tons of others, so
>         ======== Someone™ should survey that field ======
>     Also, there the usual suspects:
>     # AWS (amazon)
>     # GCE (google)
>     # heroku
>     I think all of those would need a lot of learning and drastic changes
> to our setup, even when consolidates.
>     However, they might be lots cheaper.
>
>     # last not least, rackspace.
>     We could ask them, what a consolidated setup would cost us.
>
>   - Asking People nicely
>
>     We as Squeak community could ask people or providers nicely whether
> they would host us or even sponsor us!
>
>     # netcup.de has an explicit opportunity for non-provits to be
> sponsored[7]
>     This is especially interesting, as we have a registered non-profit in
> Germany, Squeak e.V. which can accept tax-deductable donations, and that
> would fit nicely.
>     == They need a written application ===
>     I don't know whether they would sponsor wholly or partly.
>
>         ===========Please look around whether you know hosters that would
> sponser..  ============
>
> IV. The End
> ===========
>
> TL;DR: No tl;dr, please read the thing.
>
> You may panic now.
>
> Best regards
>         -Tobias
>
>
> [1]: please recall the box2 incident, where, after a service outage, we
> hat to resurrect box2 piggy-backed on box4. This setup kind-of worked, but
> as we know, isKindOf should be avoided.
> [2]: Yes, this is a lot.
> [3]: I don't want to elaborate the why's and wherefore's of the
> SFC/Rackspace thing, just how we could deal with it.
> [4]: Details:
> IAN     MEM: 10M        HDD: 10M        LOG: 1.2G
> ALAN    MEM: 100M       HDD: 53G        LOG: 1.8G       (nginx)
> ADELE   MEM: 1G         HDD: 10.5G      LOG: 1.5G
>  (postfix,amavis,clamav,mailman,nginx,fcgiwrap,postsrsd,)
> ANDREAS MEM: 200M       HDD: 55G        LOG: 300M       (source.squeak.org
> )
> DAN     MEM: 120M       HDD: 25G        LOG: 1.1G       (squeaksource.com)
> TED     MEM: 50M        HDD:  6G        LOG: 800M       (wiki,squeakmap)
> DAVID   MEM: 20M        HDD:  8G        LOG: 1.2G
>  (nginx,planet)+(old stuff)
> SCOTT   MEM: 400M       HDD:  2G        LOG: 1.2G
>  (nginx,php(mantis),codespeed)
> (MYSQL) MEM: 50M        HDD: 200M       LOG: 10M        GUESS
> OVERHEAD
>         MEM: +120M      HDD: +5G        LOG: -300M each
> (ssh,psad,fail2ban,system)
> [5]: I run my personal one, it is a nightmare.
> [6]: https://www.inwx.com/en/
> [7]: https://www.netcup.eu/ueber-netcup/public-relations.php
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/box-admins/attachments/20171006/32bd7091/attachment-0001.html>