Box-Admins October 2017

box-admins@lists.squeakfoundation.org

9 participants
10 discussions

Access to the new server(s)
by Tobias Pape 05 Apr '19

05 Apr '19

Dear all [ACTIONS AT END] with retroactive blessing of Levente, I have now prepared eight VMs on Rackspace. Here's the overview: ======================================================================================================================= Name Name (ext) intended use Unix Users Public Ports Private Ports Public IPv4 Private IPv4 ----------------------------------------------------------------------------------------------------------------------- ian ssh.squeak.org ssh-gateway ssh 1022 22 104.130.6.82 10.208.225.29 alan *.squeak.org webserver webteam 80, 443 22 104.239.229.92 10.176.200.8 adele lists.... mailinglists (tbd) 25, 587, 465 22, 8080 162.242.237.43 10.208.160.56 andreas -------- source.squeak chrismuller* -------- 22, 8080 irrelevant 10.208.161.222 dan -------- squeaksource davidlewis* -------- 22, 8080 irrelevant 10.176.197.150 ted -------- squeak wiki+map (tbd) -------- 22, 8080, 8081 irrelevant 10.176.130.111 david -------- jenkins (tbd) -------- 22, 8080 irrelevant 10.208.194.45 scott -------- misc (tbd) -------- 22, 8080, 8081 irrelevant 10.176.199.169 ======================================================================================================================= Currently, Levente and me have sudo on all these machines. Users with * also do. Note that _no_ server exposes SSH on port 22 on a public IP. This is intentional to narrow attack vectors for script kiddies. How to login? Ian is the ssh gateway so you have to connect to ian _first_ and use (1) local forwarding or (2) proxy jumping. I have installed the Public keys from most of you for the 'ssh' user on ian. Please verify by ssh -p1022 -lssh 104.130.6.82 you should see restrict shell, no commands # (you get out with crtl-d, ctrl-c, or killing ssh) How to reach the other servers? Example for 'andreas' variant (1): Do a local forward by ssh -AN -L22221:10.176.200.8:22 -p1022 -lssh 104.130.6.82 and then ssh -lYOURNAME -p22221 localhost (-N maybe optional, but then you see 'restrict shell, no commands #') Or in your .ssh/config you can put Host ian.squeak.org User ssh Hostname 104.130.6.82 Port 1022 LocalForward 222221 10.176.200.8:22 Host andreas.squeak.org User YOURNAME Hostname localhost Port 222221 And then say 'ssh -AN ian.squeak.org' and then 'ssh andreas.squeak.org' variant (2): (a) You have OpenSSH >= 7.3 Do a Jump with ssh -J ssh@104.130.6.82:1022 YOURNAME(a)10.176.200.8 Or in your .ssh/config you can put Host ian.squeak.org User ssh Hostname 104.130.6.82 Port 1022 Host andreas.squeak.org User YOURNAME Hostname 10.176.200.8 ProxyJump ian.squeak.org And then say 'ssh andreas.squeak.org' (b) You have OpenSSH >= 5.4 Do a Jump via ssh -o ProxyCommand="ssh -lssh -p1022 -W %h:%p 104.130.6.82" YOURNAME(a)10.176.200.8 Or in your .ssh/config you can put Host ian.squeak.org User ssh Hostname 104.130.6.82 Port 1022 Host andreas.squeak.org User YOURNAME Hostname 10.176.200.8 ProxyCommand ssh -W %h:%p ian.squeak.org And then say 'ssh andreas.squeak.org' (c) You have OpenSSH < 5.4 Use variant (1) We will shortly start RSYNC-ing over data from box3 and box4 as well as replicatiing DNS entries before switching over. [ACTION REQUIRED] - Who needs access to which servers? - Do we need Jenkins anymore? As always, questions appreciated. Best regards -Tobias

8 36

Re: [Box-Admins] [Board] Bandwidth
by Bert Freudenberg 24 Oct '17

24 Oct '17

Was there something in the bill, maybe? - Bert - On Tue, Oct 24, 2017 at 7:18 PM, Marcel Taeumel <marcel.taeumel(a)hpi.de> wrote: > Not quite. No sums. Just this: > > > Am 24.10.2017 17:43:14 schrieb Chris Muller <ma.chris.m(a)gmail.com>: > Does our existing hosting provide our stats? > > On Tue, Oct 24, 2017 at 8:58 AM, Marcel Taeumel wrote: > > Hi, there. > > > > I want to request hosting services from OSU [1]. For this, I have to > > estimate the bandwidth. So, can someone please look into the Rackspace > logs > > and tell me? :) > > > > Based on a community size of 1000 using our authors list, I have most > > difficulties in estimating the bandwidth usage for code updates. Diffs > vary > > from 10 kB to 200 kB. So for our 90 packages, if each touched twice a > week, > > the estimation varies from 7.8 GB/month to 156 GB/month. > > > > Help. :) > > > > Best, > > Marcel > > > > [1] http://osuosl.org/request-hosting/ > >

2 1

Re: [Box-Admins] [Board] Bandwidth
by Chris Muller 24 Oct '17

24 Oct '17

Does our existing hosting provide our stats? On Tue, Oct 24, 2017 at 8:58 AM, Marcel Taeumel <marcel.taeumel(a)hpi.de> wrote: > Hi, there. > > I want to request hosting services from OSU [1]. For this, I have to > estimate the bandwidth. So, can someone please look into the Rackspace logs > and tell me? :) > > Based on a community size of 1000 using our authors list, I have most > difficulties in estimating the bandwidth usage for code updates. Diffs vary > from 10 kB to 200 kB. So for our 90 packages, if each touched twice a week, > the estimation varies from 7.8 GB/month to 156 GB/month. > > Help. :) > > Best, > Marcel > > [1] http://osuosl.org/request-hosting/

1 0

Hangout
by Bert Freudenberg 18 Oct '17

18 Oct '17

We have problems with our regular hangout, please join this one: https://hangouts.google.com/call/TiwVn26TJXZJ-5PoocnfAAEI - Bert -

1 0

Squeak Server state of the Union or Hold Your Horses but Not Too Long
by Tobias Pape 17 Oct '17

17 Oct '17

Dear All As the primary person responsible for the current server setup and maintaining some contact with the sfc admins, I'd like to give a bit insight into the situation. I. The Past =========== Around this time last year we started migrating our services from Gandi to Rackspace. This was because (a) the Gandi service was not very responsive and (b) about to be shut down December 2016. But also (c) our server setup had been "worn out" over time[1] and (d) the SFC had for a long time offered to host the services on Rackspace. Seeing that time was about to run out, I stepped forward and, admittedly, relatively singlehandedly decided the new setup. Having the disk-shortage, interference between services and the effect of 'dragging down' other services when dying in mind, I decides to "spread out" all services as it seemed justifiable. This was mainly due to the _very_ generous offer that we were allowed to use Rackspace services up to an amount of about $2000. Therefore, I created a little network of Squeak-related services. I had separated the services into web/files, mail, squeaksource (source.squeak.org and squeaksource.com separate), mantis, others (wiki, map), and later planet.squeak and codespeed. Every server is backed up individually. We make also use of one Rackspace MySql database. The ideas here were: 1. Have only ONE ssh entrypoint to the squeak services (this is why we have the ssh gateway) 2. Isolate the services as good as possible 3. Avoid being offline when (a) a server crashes or (b) someone breaks in or (c) some admin makes a mistake. 4. Have backups of the services. 5. Have headroom on each server to be future-proof. Additionally, I had hoped that this service would continue for a long time, so that only minor adjustments would be necessary by the admins. Also, I wanted to minimize attack surface with this setup, so that one compromised service wouldn't expose others. The logs suggest that this threat is not theoretical. All that contributed to me setting up 8 servers with 1..2GB Memory, and 20..300GB HDD and 1 or 2 CPUs each (plus a Rackspace managed MySQL DB). They are all backed up by Rackspace. And, oh, our DNS is managed there, too. Since September 2016, we hat one infrastructure outage, which was the database for one hour in the middle of the night. So far so good. II. The Present =============== The setup as we use it would cost around $500–$600 per month if we had to pay it ourselves[2]. But now we most probably have to leave[3]. To be clear: I _do not_ suggest that we continue in the very setup we have and try to replicate it elsewhere. However, I think it is a good idea to _know_ what we are actually using. I tallied all services and servers and so on[4] and came to the conclusion, that we currently use the equivalent of - 2-2.5GB RAM - 200 GB Disk - 4-6 CPUs [NOTE: THESE ARE NOT EQUIVALENT TO REQUIREMENTS FOR A NEW SETUP!] These services should therefore be satisfiable with a setup <$100 month. III. The Future =============== I think that it is perfectly possible to have a setup with 2 (Two) or 3 (Three) servers. I would not suggest to consolidate everything, at least web/files and mail should be separated. But if there's no room for that, yes, we would be able to have everything on one server. That being said here are a few things to consider for the Future Squeak Infrastructure: * We need at least a month to migrate. That means, we must start end of the month THE LATEST. * We need a new DNS provider fast. I strongly advise against running a DNS server ourselves again[5]. Also, Networksolutions has a very poor user experience. I personally use INWX[6] and it works well but any other decent registrar should work. However, this is to be coordinated with Dan Ingalls and Goran Krampe, as Dan owns squeak.org. Similar stuff to be done for squeakfoundation.org. (and maybe squeakland because of mails). * We need a new hosting provider. We have two or three main ways: Pay for ourselves, ask people nicely, (and: hope the SFC figures something out in time, but I wont bet on that) - Paying for ourselves could mean several things, like renting servers, webspace, or the like. We would probably try to reduce our service footprint even further, by, eg. hosting part of the website with github.com (via pages or the like) or moving from Mantis to github issues, trello, frogbuz or so, but I don't think that would do much in terms of footprint. NOTE: I do estimates with 2 servers w/ 2-4 CPUs, 8 or 16 GB, >200GB HDD each.==== There are several options, but I personally only know a few (Euro-centric, because, you know...() # hetzner.de has been our host for some time and we would probably come around some 40-100€/month # netcup.de is what I use personally and they are a tad cheaper Those have the problem that we would have to care for backup manually (they both offer space for that, at a fee, but without automatization) I think there are tons of others, so ======== Someone™ should survey that field ====== Also, there the usual suspects: # AWS (amazon) # GCE (google) # heroku I think all of those would need a lot of learning and drastic changes to our setup, even when consolidates. However, they might be lots cheaper. # last not least, rackspace. We could ask them, what a consolidated setup would cost us. - Asking People nicely We as Squeak community could ask people or providers nicely whether they would host us or even sponsor us! # netcup.de has an explicit opportunity for non-provits to be sponsored[7] This is especially interesting, as we have a registered non-profit in Germany, Squeak e.V. which can accept tax-deductable donations, and that would fit nicely. == They need a written application === I don't know whether they would sponsor wholly or partly. ===========Please look around whether you know hosters that would sponser.. ============ IV. The End =========== TL;DR: No tl;dr, please read the thing. You may panic now. Best regards -Tobias [1]: please recall the box2 incident, where, after a service outage, we hat to resurrect box2 piggy-backed on box4. This setup kind-of worked, but as we know, isKindOf should be avoided. [2]: Yes, this is a lot. [3]: I don't want to elaborate the why's and wherefore's of the SFC/Rackspace thing, just how we could deal with it. [4]: Details: IAN MEM: 10M HDD: 10M LOG: 1.2G ALAN MEM: 100M HDD: 53G LOG: 1.8G (nginx) ADELE MEM: 1G HDD: 10.5G LOG: 1.5G (postfix,amavis,clamav,mailman,nginx,fcgiwrap,postsrsd,) ANDREAS MEM: 200M HDD: 55G LOG: 300M (source.squeak.org) DAN MEM: 120M HDD: 25G LOG: 1.1G (squeaksource.com) TED MEM: 50M HDD: 6G LOG: 800M (wiki,squeakmap) DAVID MEM: 20M HDD: 8G LOG: 1.2G (nginx,planet)+(old stuff) SCOTT MEM: 400M HDD: 2G LOG: 1.2G (nginx,php(mantis),codespeed) (MYSQL) MEM: 50M HDD: 200M LOG: 10M GUESS OVERHEAD MEM: +120M HDD: +5G LOG: -300M each (ssh,psad,fail2ban,system) [5]: I run my personal one, it is a nightmare. [6]: https://www.inwx.com/en/ [7]: https://www.netcup.eu/ueber-netcup/public-relations.php

4 3

[INFO] I restarted squeaksource.com
by David T. Lewis 08 Oct '17

08 Oct '17

Something happened to the squeaksource.com image that made it unresponsive with high CPU usage, probably beginning a day ago. I could not successfully connect through VNC to resolve, so I restarted the image. I think that one new account was lost for someone doing "PharoExercices", so probably a classroom exercise of some sort. I most recent updates on disk were: -rw-r--r-- squeaksourcecom/www-data 6012 2017-10-06 23:34 ./Cryptography/Session-hh.30(29).mcd -rw-r--r-- squeaksourcecom/www-data 2182 2017-10-06 18:31 ./PharoExercices/Counter-ChristopheCaulier.7.mcz -rw-r--r-- squeaksourcecom/www-data 1906 2017-10-06 17:55 ./PharoExercices/Counter-ChristopheCaulier.4.mcz -rw-r--r-- squeaksourcecom/www-data 1394 2017-10-06 16:32 ./PharoExercices/Counter-ChristopheCaulier.2.mcz Cryptography/Session-hh.30 is still present in the restarted image, but CristopheCaulier is not. Dave

1 1

Amazon Route 53 dns
by PAUL DEBRUICKER 06 Oct '17

06 Oct '17

I could help set DNS up with Amazon's Route 53 DNS service. I've been using it for about four years. Its $0.50 per month per zone (e.g. squeak.org & wiki.squeak.org are the same 'zone' but not squeaksource.com) and $0.40 per million DNS queries. The query usage is shared across zones. page: https://aws.amazon.com/route53/ pricing: https://aws.amazon.com/route53/pricing/ faq: https://aws.amazon.com/route53/faqs/

1 0

Squeak Server state of the Union or Hold Your Horses but Not Too Long
by Chris Cunnington 05 Oct '17

05 Oct '17

I think the community has learned that free servers always become more trouble than their worth. Therefore, the clearest path forward is to get two boxes at Hetzner, cut services and raise money. Some details are vexing (whose name is the server in? which services get cut? where is the money coming from?). But I’d be surprised if there were a more clear path forward. FWIW, Chris

1 0

SqueakSource problem
by David T. Lewis 05 Oct '17

05 Oct '17

I am currently unable to log in to http://squeaksource.com, which is showing a password violation. I also do not have write access to the repository from Monticello in Squeak for my account (dtl). I do not have confirmation yet, but I assume this is a problem that affects all users, and that squeaksource.com is effectively read-only right now. I do not know the cause of the problem but I'm reporting it on box-admins now and will follow up if and when I figure out what went wrong. Here is what happened: A day or so ago I created a new project (EtoysProject) on SqueakSource. Today I looked at it, and the new project was gone, and I could see that a new account registration for Marcel had also disappeared. Looking on dan.box.squeak.org, I could see that the squeaksource image had automatically restarted about 35 hours earlier, before I had created the new project. Probably this is unrelated to the current problem, and would have been caused by the image running out of file handles. This is a known bug, and I have a monitor process in the image that watches for it and restarts the image if it happens. The ss.log file shows projects being recovered some time after my new project changes. This seems to happen every once in a while, and I do not know the reason. But in this case it may indicate something going wrong that may have led to the lost EtoysProject information. The lost project information was in the image, but the MCZ file for this project was still on disk. After seeing the above problem, I re-created the EtoysProject project through the web interface, reloaded the MCZ file (so the image would know about it, even though it was on disk already), and verified that the MCZ was now visible in the archive from a Squeak MC browser. I then tried to make a backup of the image in its updated stated. I connected through VNC, saved the image interactively, and made a backkup copy of it in the ./BACKUPS directory. After this last step, I checked http://squeaksource.com and found that I could no longer log in as 'dtl'. I tried copying an MCZ from a Squeak image to SqueakSource, and got an access violation there as well. I do not know what is wrong, but it appears to have happened as a side effect of my saving the image through a VNC session. The last known good backup of the squeaksource.com image (tested and verified today on my home PC) is: dan.box.squeak.org://srv/squeaksourcecom/SqueakSource/BACKUPS/ss-image-save-20170917.tgz This backup was taken about 2 1/2 weeks ago, and if necessary, squeaksource.com may be restarted from this image. But this would result in any new project definitions or user account changes being lost (all the disk-resident MCZ files will still be present). I am taking no further action at this time, just trying to figure out what went wrong, and I am writing this up so we have a record of the problem. Dave

1 1

Re: [Box-Admins] [squeak-dev] Ideas for web hosting for squeak.org?
by Ron Teitelbaum 04 Oct '17

04 Oct '17

On Wed, Oct 4, 2017 at 4:48 PM, tim Rowledge <tim(a)rowledge.org> wrote: > > > On 04-10-2017, at 1:40 PM, Ron Teitelbaum <ron(a)usmedrec.com> wrote: > > > > Hey Tim, > > <snip/> > The Rackspace charges are shown as an incredible $560/month, which seems > quite insane. I guess if you’re donating you can put whatever number you > like and use that on your tax forms… > That is insane! We run a lot of servers at AWS and the most expensive one is around $2500 a year. > > > What are the server requirements? CPU, Disk Space, Operating System? > How much are the normal fees for Rackspace? > > We need to get the box admins and web teams to tell us about that. I added Box Admins. I took this off the public list so we could disucss it privately. Hello Box Admins, Can you let me know if this would be good enough? https://aws.amazon.com/ec2/instance-types/ vCPU Memory (GiB) Storage (GB) Networking Performance Physical Processor Clock Speed (GHz) Intel AVX† <https://aws.amazon.com/ec2/instance-types/#intel> Intel AVX2† Intel Turbo <https://aws.amazon.com/ec2/instance-types/#intel> EBS OPT <https://aws.amazon.com/ec2/instance-types/#ebs> Enhanced Networking† c3.large 2 3.75 2 x 16 SSD Moderate Intel Xeon E5-2680 v2 2.8 Yes - Yes - Yes We would add extra disk drive space so would need to know how much and what OS is required. I normally run on CentOS 6.7 but could also do Ubuntu if for some reason that would be better. Ron > > tim > -- > tim Rowledge; tim(a)rowledge.org; http://www.rowledge.org/tim > Useful Latin Phrases:- Non curo. Si metrum non habet, non est poema = I > don't care. If it doesn't rhyme, it isn't a poem. > > > >

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Box-Admins October 2017