<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi,<div class=""><br class=""></div><div class="">I've been running a Squeak web server on the wide-open internet for over a year now (WebClient/WebServer/Seaside/nginx proxy). Within a few days it will always stop answering the phone, begin eating ~99% of the CPU, and eventually segfault (though it has become less segfaulty on this May 2020 VM). I've tried the recommended Unix tweaks for it (file handle limits, etc) but I believe there is something going on inside the image. Today's issue may be related (at least) to some code in WebMessage.</div><div class=""><br class=""></div><div class="">tl;dr:</div><div class=""><br class=""></div><div class=""><div class="">I'm proposing a change to WebMessage>>#streamDirectlyFrom:to:size:progress: </div><div class="">...but my proposed change could very well be wrong, or be poor logic. So I'm writing up my lengthy rationale below.</div></div><div class=""><br class=""></div><div class="">What do you think about changing:</div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>[(sizeOrNil == nil and:[srcStream atEnd not]) or:[totalRead < size]] whileTrue:[</div><div class=""><br class=""></div><div class="">to:</div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>[sizeOrNil == nil or: [totalRead < size and: [srcStream atEnd not]]] whileTrue:[</div><div class=""><br class=""></div><div class="">?</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">My logic could be wrong, and I don't understand the sizeOrNil check here. But after I made this change, the infinite loop went away, GC destroyed hundreds of stale Sockets and Processes, and my Seaside app began answering again (note that this is the first time I've been successful at bringing it back to life). </div><div class=""><br class=""></div><div class="">If the original logic is correct as intended, it seems to be a trade-off between continuing to read from the stream if we haven't yet read #size characters, versus not trying to read from a stream that is #atEnd. My version breaks out of the loop if the stream is #atEnd even if the method believes there are characters left to read. The reason I think this is worthwhile, is that #totalRead will (endlessly) be 0 if the Socket underneath the SocketStream is closed, and there are no more bytes to read, thus causing an infinite loop. It is *possible* (and mentioned below) that the design is supposed to catch a #ConnectionClosed exception, but in my case one never got thrown. (note the Socket's state was #otherEndClosedButNotThisEnd)</div><div class=""><br class=""></div><div class="">It's also possible that the method is <i class="">designed</i> to continue reading from the SocketStream in case it is expecting more data which just hasn't arrived yet (as I think I saw in a method comment somewhere), but this may not make sense when the underlying Socket has been closed and the method keeps reading zero characters from it. </div><div class=""><br class=""></div><div class="">Of course, actually, <i class="">anything</i> is possible. What do <i class="">I</i> know? :D</div><div class=""><br class=""></div><div class="">The rest of this message contains my exploration, debugging session, and further rationale. </div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">a Tim</div><div class=""><br class=""></div><div class="">-------</div><div class=""><br class=""></div><div class="">After running for eight days, Squeak had 300 socket connections open. This is far too many for my very, very unpopular website:</div><div class=""><br class=""></div><div class="">$ sudo netstat -a --program | awk '/http.*squeak/ { print $6 }' | sort | uniq -c<br class=""> 297 CLOSE_WAIT<br class=""> 6 ESTABLISHED<br class=""> 1 LISTEN</div><div class=""><br class=""></div><div class="">A scan of Processes belonging to WebServer returned over 500, and climbing:</div><div class=""><br class=""></div><div class="">Process allSubInstances select: [:proc | <br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>proc suspendedContext<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>ifNil: [ false ]<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>ifNotNil: [:context | <br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>context closure<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>ifNil: [ false ]<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>ifNotNil: [:closure | | receiver |<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>receiver := closure outerContext receiver.<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>(receiver respondsTo: #outerContext)<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>ifTrue: [ receiver outerContext receiver class == WebServer ] <br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>ifFalse: [ false ] ] ] ]<br class=""><br class=""></div><div class=""><br class=""></div><div class="">Note that none of them are terminated:</div><div class=""><br class=""></div><div class="">Bag newFrom: (self collect: [:proc | proc isTerminated])<br class="">-> a Dictionary(false->507 true->1 )</div><div class=""><br class=""></div><div class=""><div class="">Most/all of them seem to be hanging out in this method:</div><div class=""><br class=""></div><div class="">method: <span class="Apple-tab-span" style="white-space: pre;"> </span>(BlockClosure>>#newProcess "a CompiledMethod(3871190)")<br class="">closureOrNil: <span class="Apple-tab-span" style="white-space: pre;"> </span>[closure] in BlockClosure>>newProcess<br class="">receiver: <span class="Apple-tab-span" style="white-space: pre;"> </span>[closure] in <b class="">WebServer>>asyncHandleConnectionFrom:</b></div></div><div class=""><br class=""></div><div class=""><div class="">It seems that WebServer uses SocketStream on a Socket. So how many SocketStreams exist and where are they?</div><div class=""><br class=""></div><div class="">Bag newFrom: (SocketStream allInstances collect: [:ea | ea socket statusString asSymbol])</div><div class="">-> a Dictionary(#invalidSocketHandle->32 #otherEndClosedButNotThisEnd->1 )</div></div><div class=""><br class=""></div><div class="">So, there's one SocketStream that's closed but we're holding on to it. Why?</div><div class=""><br class=""></div><div class="">I opened a Process Browser, turned on CPU Watcher, turned on Auto-Update and found WebServer's listener process eating > 90% CPU. I opened a debugger on it. It seemed it was in an infinite loop centered around WebMessage>>streamDirectlyFrom:to:size:progress: on this very request.</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">[] in SocketStream>>beSignalingWhile:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">BlockClosure>>ensure:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">SocketStream>>beSignalingWhile:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">[] in SocketStream>>next:into:startingAt:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">BlockClosure>>on:do:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">SocketStream>>next:into:startingAt:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">WebRequest(WebMessage)>>streamDirectlyFrom:to:size:progress:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">WebRequest(WebMessage)>>streamFrom:to:size:progress:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">[] in WebRequest(WebMessage)>>getContentWithProgress:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">ByteString class(SequenceableCollection class)>>new:streamContents:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">WebRequest(WebMessage)>>getContentWithProgress:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">WebRequest(WebMessage)>>getContent</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">WebRequest(WebMessage)>>content</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">WAWebServerAdaptor>>requestBodyFor:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">WAWebServerAdaptor(WAServerAdaptor)>>requestFor:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">WAWebServerAdaptor(WAServerAdaptor)>>contextFor:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">WAWebServerAdaptor(WAServerAdaptor)>>process:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">WAWebServerAdaptor>>process:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">MessageSend>>valueWithArguments:</div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">WebServer>>invokeAction:request:</div></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><br class=""></div><div class="">Some exploit attempt sending an HTTP POST request to /tmUnblock.cgi and with a Content-Length of 227 was causing WebMessage to lose its mind. (This is when I realized that my nginx proxy configuration might be wrong, and some obvious exploit attempts could be getting through to Squeak. But that's a different matter.)</div><div class=""><br class=""></div><div class="">Now I believe any of the following could have been happening:</div><div class=""><br class=""></div><div class=""><div class="">* WebMessage & friends seem to take the supplied Content-Length header in an HTTP request as being literal and true, even though a client could be supplying a wrong or misleading value. I don't know if there are tests against this but I might write one.</div></div><div class=""><br class=""></div><div class="">* If the Socket is closed, the code on the stack trace above will keep trying to read from it. I can't tell if:</div><div class=""><br class=""></div><div class="">a) there is an arithmetic problem somewhere, or</div><div class="">b) if #ConnectionClosed isn't being raised or caught in the proper way, or </div><div class="">c) if there's a logic error in WebMessage>>streamDirectlyFrom:to:size:progress: , or </div><div class="">d) this is totally intentional, or</div><div class="">e) all of the above, none of the above, or something else. </div><div class=""><br class=""></div><div class="">I opted to fix (c).</div><div class=""><br class=""></div><div class="">* This portion of WebMessage>>streamDirectlyFrom:to:size:progress: is troublesome to me, and my change seems to make the infinite loop end:</div><div class=""><br class=""></div><div class=""><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>[(sizeOrNil == nil and:[srcStream atEnd not]) or:[totalRead < size]] whileTrue:[</div></div><div class=""><br class=""></div><div class="">In this case, totalRead was always less than size, because the connection was closed and the methods would always return empty strings for what had been read (thus, totalRead = 0). So, [totalRead < size] was always True, which would override the check for [srcStream atEnd not] and it would continue to try reading from a SocketStream on a closed Socket.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div></body></html>