Christoph Thiede uploaded a new version of Network to project The Trunk:
http://source.squeak.org/trunk/Network-ct.269.mcz
==================== Summary ====================
Name: Network-ct.269
Author: ct
Time: 19 May 2023, 1:50:14.181195 pm
UUID: f57a07c4-63dc-7442-8bfe-57aeb8de3682
Ancestors: Network-ct.268
Adds support for ascdate/cdate format in MailMessage headers. While this is not covered by RFC822, some mboxes do contain this format, such as the older years of http://lists.squeakfoundation.org/pipermail/squeak-dev/. Also fixes offset of fallback time.
=============== Diff against Network-ct.268 ===============
Item was changed:
----- Method: MailMessage>>time (in category 'accessing') -----
time
| dateField |
dateField := (self fieldNamed: 'date' ifAbsent: [ ^0 ]) mainValue.
+ ^ [self timeFrom: dateField] ifError: [:err :rcvr | (Date today - (Date newDay: 1 year: 1980)) asSeconds].!
- ^ [self timeFrom: dateField] ifError: [:err :rcvr | Date today asSeconds].
- !
Item was changed:
----- Method: MailMessage>>timeFrom: (in category 'parsing') -----
timeFrom: aString
"Parse the date and time (rfc822) and answer the result as the number of seconds
since the start of 1980."
| s t rawDelta delta plusOrMinus |
s := ReadStream on: aString.
"date part"
+ t := ((self readDateFrom: s) ifNil: [Date today]).
+
+ (aString size > 5 and: [(aString atLast: 5) = $ and: [aString last isDigit and: [(aString atLast: 2) isDigit] and: [(aString atLast: 3) isDigit] and: [(aString atLast: 4) isDigit]]]) ifTrue: [
+ "ctime format - not covered by rfc822, but used in mboxes, such as the older years of http://lists.squeakfoundation.org/pipermail/squeak-dev/"
+ | year |
+ year := Integer readFrom: (aString last: 4) readStream.
+ year >= 1900 "not just a timezone without trailing +" ifTrue:
+ [t := t dayMonthYearDo: [:d :m :y | Date year: year month: m day: d].
+ [s peekBack isDigit] whileTrue: [s skip: -1].
+ s := (aString allButLast: 5) readStream position: s position; yourself]].
+
+ t := t asSeconds.
+
- t := ((self readDateFrom: s) ifNil: [Date today]) asSeconds.
-
[s atEnd or: [s peek isAlphaNumeric]]
whileFalse: [s next].
"time part"
s atEnd ifFalse: ["read time part (interpreted as local, regardless of sender's timezone)"
(s peek isDigit) ifTrue: [t := t + (Time readFrom: s) asSeconds].
].
s skipSeparators.
"Check for a numeric time zone offset"
('+-' includes: s peek) ifTrue:
[plusOrMinus := s next.
rawDelta := (s peek isDigit) ifTrue: [Integer readFrom: s] ifFalse: [0].
delta := (rawDelta // 100 * 60 + (rawDelta \\ 100)) * 60.
t := plusOrMinus = $+ ifTrue: [t - delta] ifFalse: [t + delta]].
"We ignore text time zone offsets like EST, GMT, etc..."
^ t - (Date newDay: 1 year: 1980) asSeconds
"MailMessage new timeFrom: 'Thu, 22 Jun 2000 14:17:47 -500'"
"MailMessage new timeFrom: 'Thu, 22 Jun 2000 14:17:47 --500'"
"MailMessage new timeFrom: 'on, 04 apr 2001 14:57:32'"!
Christoph Thiede uploaded a new version of Network to project The Trunk:
http://source.squeak.org/trunk/Network-ct.268.mcz
==================== Summary ====================
Name: Network-ct.268
Author: ct
Time: 19 May 2023, 1:40:42.341195 pm
UUID: 63ac44a4-d515-6742-bc5e-00724d671c09
Ancestors: Network-tpr.267
Fixes parsing of multipart documents. As per RFC1341, the final separator can - or even should - be extended by two dashes. Note that we do not properly ignore preamble and epilogue for now, simply because of lack of need.
=============== Diff against Network-tpr.267 ===============
Item was changed:
----- Method: MIMEDocument class>>parseParts:withSeparator: (in category 'instance creation') -----
parseParts: bodyText withSeparator: separator
"private -- parse the parts of the message and store them into a collection"
| parseStream msgStream messages normalizedSeparator |
parseStream := ReadStream on: bodyText.
msgStream := LimitingLineStreamWrapper on: parseStream delimiter: separator.
normalizedSeparator := separator asLowercase.
msgStream limitingBlock: [:aLine | | normalizedLine |
normalizedLine := aLine withoutTrailingBlanks asLowercase.
normalizedLine = normalizedSeparator or: "Match the separator"
+ [normalizedLine = ('--',normalizedSeparator)] or: "or the separator after --"
+ [normalizedLine = (normalizedSeparator, '--')] or: "or the final separator before --"
+ [normalizedLine = ('--',normalizedSeparator, '--')]]. "or the final separator before and after --"
- [normalizedLine = ('--',normalizedSeparator)] or: "or -- and the separator"
- [normalizedLine = (normalizedSeparator, '--')]]. "or the final separator with --"
"Throw away everything up to and including the first separator"
msgStream upToEnd.
msgStream skipThisLine.
"Extract each of the multi-parts as strings"
messages := OrderedCollection new.
[parseStream atEnd]
whileFalse:
[messages add: msgStream upToEnd.
msgStream skipThisLine].
^ messages collect: [:e | MIMEDocument fromPartString: e]!
Marcel Taeumel uploaded a new version of Tools to project The Trunk:
http://source.squeak.org/trunk/Tools-mt.1204.mcz
==================== Summary ====================
Name: Tools-mt.1204
Author: mt
Time: 16 May 2023, 10:52:20.408174 am
UUID: 2cd6d83a-7746-5942-b1e3-616b0ae9fc2d
Ancestors: Tools-mt.1203
In TreeBrowser, fixes performance bug in #expandClassTree. Also expand all non-extension protocols automatically.
=============== Diff against Tools-mt.1203 ===============
Item was changed:
----- Method: TreeBrowser>>expandClassTree (in category 'class tree - support') -----
expandClassTree
"Expand full class hierarchy for specific categories, top-level only for the '-- all --' category."
self isShowingAllClasses
ifTrue: [self changed: #expandNodeRequested with: {#classChildren:. ProtoObject}]
ifFalse: [(self classTreeExpanded not and: [lastClassList size <= self class classTreeAutoExpandLimit])
+ ifTrue: [self changed: #expandAllNodesRequested with: #classChildren:]]!
- ifTrue: [self classRoots do: [:ea | self changed: #expandAllNodesRequested with: #classChildren:]]]!
Item was added:
+ ----- Method: TreeBrowser>>expandMessageCategoryTree (in category 'message category tree - support') -----
+ expandMessageCategoryTree
+ "Expand all non-extension groups. Typically, there are only a few groups so browsing should be possible without having to filter or manually expand non-extension groups."
+
+ self messageCategoryRoots do: [:ea |
+ ea first ~= $- "not -- all --, -- core --, -- extensions --"
+ ifTrue: [self changed: #expandAllNodesRequested with: {#messageCategoryChildren:. ea}]]!
Item was changed:
----- Method: TreeBrowser>>updateMessageCategoryTree: (in category 'updating') -----
updateMessageCategoryTree: newList
lastMessageCategoryList := newList.
+ self changed: #messageCategoryRoots.
+ self expandMessageCategoryTree.!
- self changed: #messageCategoryRoots.!
Marcel Taeumel uploaded a new version of Tools to project The Trunk:
http://source.squeak.org/trunk/Tools-mt.1203.mcz
==================== Summary ====================
Name: Tools-mt.1203
Author: mt
Time: 16 May 2023, 10:11:05.470174 am
UUID: c4f5d92e-3df6-374e-8bd0-f52038816bb7
Ancestors: Tools-mt.1202
In TreeBrowser, tweak the behavior of class-tree auto-expansion and whether to show collapse/expand arrows or not. Compact hierarchies benefit from not showing the arrows for less clutter.
Note that you can expand all children via SHIFT+RightArrow or SHIFT+RedClick.
=============== Diff against Tools-mt.1202 ===============
Item was added:
+ ----- Method: TreeBrowser class>>classTreeAutoExpandLimit (in category 'defaults') -----
+ classTreeAutoExpandLimit
+ "Answer the maximum number of classes where the class tree should be expanded automatically."
+
+ ^ 750!
Item was added:
+ ----- Method: TreeBrowser class>>classTreeAutoExpandLockLimit (in category 'defaults') -----
+ classTreeAutoExpandLockLimit
+ "Answer the maximum number of classes where the class tree should be expanded automatically and the user should not be able to collapse the tree but just use the type-in filter. This number should be smaller than #classTreeAutoExpandLimit."
+
+ ^ 50!
Item was changed:
----- Method: TreeBrowser>>classTreeExpanded (in category 'class tree') -----
classTreeExpanded
"For reasons of usability and performance, we do not expand the entire class tree when potentially all classes in the system would be shown."
+ ^ (self isShowingAllClasses or: [lastClassList size > self class classTreeAutoExpandLockLimit]) not!
- ^ self isShowingAllClasses not!
Item was changed:
----- Method: TreeBrowser>>expandClassTree (in category 'class tree - support') -----
expandClassTree
"Expand full class hierarchy for specific categories, top-level only for the '-- all --' category."
self isShowingAllClasses
+ ifTrue: [self changed: #expandNodeRequested with: {#classChildren:. ProtoObject}]
+ ifFalse: [(self classTreeExpanded not and: [lastClassList size <= self class classTreeAutoExpandLimit])
+ ifTrue: [self classRoots do: [:ea | self changed: #expandAllNodesRequested with: #classChildren:]]]!
- " ifFalse: [self classRoots do: [:ea | self changed: #expandAllNodesRequested with: #classChildren:]]"
- ifTrue: [self changed: #expandNodeRequested with: {#classChildren:. ProtoObject}].!
Christoph Thiede uploaded a new version of WebClient-Core to project The Treated Inbox:
http://source.squeak.org/treated/WebClient-Core-ct.126.mcz
==================== Summary ====================
Name: WebClient-Core-ct.126
Author: ct
Time: 12 October 2020, 7:05:32.190311 pm
UUID: a64ca560-814a-f940-8f64-e66a25cedc61
Ancestors: WebClient-Core-ul.123
Proposal to implement pre-authentication on WebClient.
MOTIVATION.
Until now, the authentication flow in Squeak's WebClient looks like this:
First, a request is made without trying to authenticate the user. If the request fails with an error 401 (Unauthorized) or an error 407 (Proxy Authentication Required), the authentication headers are added to the request, and the request is retried.
However, this does not work properly in some situations.
For example, many modern REST APIs use to return an error 404 if an attempt is made to access a private resource without authenticating before [1] which currrently makes it impossible to authenticate to these APIs using the WebClient.
Another issue I encountered today lies in some particular servers not requesting a specific authentication method via the WWW-Authenticate header along a 401 response as specified by the protocol [2]. Concretely, I encountered this problem with the quite popular GitHub API so I think our client should be robust enough to handle this contract violation.
APPROACH.
This patch adds a new property for the #preAuthenticationMethod to the WebClient class. It can be set to a symbol indicating any authentication method that is supported by the WebClient, e.g. #basic or #bearer. (Digest access authentication, however, cannot be used at this place because it depends on a realm specified by the server.)
If this property is set, the relevant authentication headers will be added to the request already before the first attempt is made to request the resource.
In addition, the patch refactors and reformatst the methods #authenticate:from: and #sendRequest:contentBlock:.
The TESTS work as well as always (a number of them failing sporadically, but after some trials, I get a green bar again).
WHAT REMAINS TO BE DONE.
I deleted the fixme "Pre-authenticate the request if we have valid auth credentials" comment which I think was exactly what I implemented in this patch. I hope this assumption was correct? Also, another fixme comment requests to preserve the authState after following a redirect. Instead, with this patch, any specified pre-authentication method will be reused after every redirect. I did not fix this because I do not have a use-case scenario for it. Can we leave this as-is, and in a future version, could we simply delete this send to #flushAuthState?
Also, I'm not sure about whether pre-authentication maybe should be the default for every request containing a username/password specification. This would speed up every web request that uses credentials by up to the factor 2 because we could save one futile query. Also, it appears to be the state-of-the-art solution, popular tools such as curl specify the credentials in the first run already. On the contrary, it would be a breaking change, and looking at the comment in #preAuthenticationMethod, pre-authentication as an opt-out feature might break or at least slow down NTLM/Negotiate use cases. However, I never heard of this before. Are these protocols still relevant at all?
Please give this patch a careful review because still, all knowledge I have about this domain is collected from StackOverflow and Wikipedia a few hours ago.
REFERENCES.
[1] https://stackoverflow.com/a/17688080/13994294
[2] https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401
=============== Diff against WebClient-Core-ul.123 ===============
Item was changed:
Object subclass: #WebClient
+ instanceVariableNames: 'flags server scheme timeout stream cookies proxyServer lastScheme lastServer lastPort maxRedirect redirections userAgent authParams proxyParams accessLog debugLog preAuthenticationMethod'
- instanceVariableNames: 'flags server scheme timeout stream cookies proxyServer lastScheme lastServer lastPort maxRedirect redirections userAgent authParams proxyParams accessLog debugLog'
classVariableNames: 'DebugLog FlagAcceptCookies FlagAllowAuth FlagAllowRedirect ProxyHandler'
poolDictionaries: ''
category: 'WebClient-Core'!
!WebClient commentStamp: 'ar 5/4/2010 13:17' prior: 0!
WebClient provides a simple yet complete HTTP client implementation.
To view the documentation evaluate:
HelpBrowser openOn: WebClientHelp.
!
Item was added:
+ ----- Method: WebClient>>authProcess:from:header:params: (in category 'authentication') -----
+ authProcess: request from: response header: authHeader params: params
+ "Process an authentication header.
+ Answer true if an authentication response could be generated; otherwise, false."
+
+ self
+ authDispatch: request
+ from: response
+ header: authHeader
+ params: params.
+
+ params at: #authResponse ifAbsent: [^ false].
+
+ "If we generated an authentication response for the header use it"
+ request
+ headerAt: ((response ifNotNil: [response code = 401] ifNil: [true])
+ ifTrue: ['Authorization']
+ ifFalse: ['Proxy-Authorization'])
+ put: (params at: #authMethod), ' ', (params at: #authResponse).
+
+ ^ true!
Item was changed:
----- Method: WebClient>>authenticate:from: (in category 'sending') -----
authenticate: request from: response
"Authenticate after having received a 401/407 response.
Returns true if we should retry, false if we fail here."
+ | headers params |
+
- "NOTE: The first time through we do NOT ask for credentials right away.
- Some authentication mechanisms (NTLM/Negotiate) can use the credentials
- of the currently logged on user. Consequently we only ask for credentials
- if we're unable to do so without asking. Methods that require credentials
- (basic, digest) test for their existence explicitly."
-
- | headers authHeader params |
-
"Pick the right set of parameters"
+ response code = 401
+ ifTrue: [
+ params := authParams.
+ headers := response headersAt: 'WWW-Authenticate'.
+ "If the connection was closed, we need to flush the
+ proxy params or we won't pick up prior credentials."
+ self isConnected
+ ifFalse: [self flushAuthState: proxyParams]]
+ ifFalse: [
+ params := proxyParams.
+ headers := response headersAt: 'Proxy-Authenticate'].
+
- response code = 401 ifTrue:[
- params := authParams.
- headers := response headersAt: 'WWW-Authenticate'.
- "If the connection was closed, we need to flush the
- proxy params or we won't pick up prior credentials."
- self isConnected
- ifFalse:[self flushAuthState: proxyParams].
- ] ifFalse:[
- params := proxyParams.
- headers := response headersAt: 'Proxy-Authenticate'.
- ].
-
"Remove any old response"
+ params removeKey: #authResponse ifAbsent: [].
+
- params removeKey: #authResponse ifAbsent:[].
-
"Process the authentication header(s)"
+ headers
+ detect: [:authHeader |
+ self
+ authProcess: request
+ from: response
+ header: authHeader
+ params: params]
+ ifFound: [:authHeader | ^ true].
+
+ "If we fall through here this can have two reasons: One is that we don't have a suitable authentication method. Check for that first."
+ params at: #authMethod ifAbsent: [^ false].
+
+ "The other possibility is that the credentials are wrong. Clean out the previous auth state and go ask for credentials."
- 1 to: headers size do:[:i|
- authHeader := headers at: i.
- self authDispatch: request from: response header: authHeader params: params.
- "If we generated an authentication response for the header use it"
- params at: #authResponse ifPresent:[:resp|
- request headerAt: (response code = 401
- ifTrue:['Authorization']
- ifFalse:['Proxy-Authorization'])
- put: (params at: #authMethod), ' ', resp.
- ^true].
- ].
-
- "If we fall through here this can have two reasons: One is that we don't have
- a suitable authentication method. Check for that first."
- params at: #authMethod ifAbsent:[^false].
-
- "The other possibility is that the credentials are wrong.
- Clean out the previous auth state and go ask for credentials."
self flushAuthState: params.
+
-
"Clean out old authentication headers"
response code = 401
+ ifTrue: [request removeHeader: 'Authorization'].
- ifTrue:[request removeHeader: 'Authorization'].
"Always clean out the proxy auth header since we don't support pre-authentication"
request removeHeader: 'Proxy-Authorization'.
+
-
"Signal WebAuthRequired"
(WebAuthRequired client: self request: request response: response)
+ signal == true ifFalse: [^ false].
+
- signal == true ifFalse:[^false].
-
"And retry with the new credentials"
+ ^ self authenticate: request from: response!
- ^self authenticate: request from: response!
Item was added:
+ ----- Method: WebClient>>preAuthenticationMethod (in category 'accessing') -----
+ preAuthenticationMethod
+ "The authentication method to be used for initial requests. Symbol, e.g. #basic or #bearer. If nil, no authentication will be used until the server requests an authentication.
+
+ NOTE: Some authentication mechanisms (NTLM/Negotiate) can use the credentials of the currently logged on user. Consequently, by default we only ask for credentials if we're unable to do so without asking."
+
+ ^ preAuthenticationMethod!
Item was added:
+ ----- Method: WebClient>>preAuthenticationMethod: (in category 'accessing') -----
+ preAuthenticationMethod: aSymbol
+ "The authentication method to be used for initial requests. See #preAuthenticationMethod."
+
+ preAuthenticationMethod := aSymbol!
Item was changed:
----- Method: WebClient>>sendRequest:contentBlock: (in category 'sending') -----
sendRequest: request contentBlock: contentBlock
"Send an http request"
| response repeatRedirect repeatAuth |
-
- "XXXX: Fixme. Pre-authenticate the request if we have valid auth credentials"
-
redirections := Dictionary new.
["The outer loop handles redirections"
+ repeatRedirect := false.
+
+ "Always update the host header due to redirect"
+ request headerAt: 'Host' put: server.
+
+ self preAuthenticationMethod ifNotNil: [:authMethod |
+ self
+ authProcess: request
+ from: nil
+ header: authMethod asString capitalized
+ params: authParams].
+
- repeatRedirect := false.
-
- "Always update the host header due to redirect"
- request headerAt: 'Host' put: server.
-
["The inner loop handles authentication"
+ repeatAuth := false.
+
+ "Connect can fail if SSL proxy CONNECT is involved"
+ self connect ifNotNil: [:resp| ^ resp].
+
+ "Write the request to the debugLog if present"
+ debugLog ifNotNil: [self writeRequest: request on: debugLog].
+
+ "Send the request itself"
+ self writeRequest: request on: stream.
+ contentBlock value: stream.
+
+ response := request newResponse readFrom: stream.
+ response url: scheme, '://', server, request rawUrl.
+
+ debugLog ifNotNil: [
+ response writeOn: debugLog.
+ debugLog flush].
+ response setCookiesDo: [:cookie|
+ self acceptCookie: cookie host: self serverUrlName path: request url].
+ accessLog ifNotNil: [
+ WebUtils logRequest: request response: response on: accessLog].
+ "Handle authentication if needed"
+ (self allowAuth and: [response code = 401 or: [response code = 407]]) ifTrue: [
+ "Eat up the content of the previous response"
+ response content.
+ repeatAuth := self authenticate: request from: response].
+
+ repeatAuth
+ ] whileTrue.
- repeatAuth := false.
-
- "Connect can fail if SSL proxy CONNECT is involved"
- self connect ifNotNil:[:resp| ^resp].
+ "Flush previous authState.
+ XXXX: Fixme. authState must be preserved for pre-authentication of requests."
+ self flushAuthState.
+
+ "Handle redirect if needed"
+ (self allowRedirect and: [response isRedirect]) ifTrue:[
- "Write the request to the debugLog if present"
- debugLog ifNotNil:[self writeRequest: request on: debugLog].
-
- "Send the request itself"
- self writeRequest: request on: stream.
- contentBlock value: stream.
-
- response := request newResponse readFrom: stream.
- response url: (scheme, '://', server, request rawUrl).
-
- debugLog ifNotNil:[
- response writeOn: debugLog.
- debugLog flush.
- ].
- response setCookiesDo:[:cookie|
- self acceptCookie: cookie host: self serverUrlName path: request url.
- ].
- accessLog ifNotNil:[
- WebUtils logRequest: request response: response on: accessLog
- ].
- "Handle authentication if needed"
- (self allowAuth and:[response code = 401 or:[response code = 407]]) ifTrue:[
"Eat up the content of the previous response"
response content.
+ repeatRedirect := self redirect: request from: response].
+ repeatRedirect
+ ] whileTrue: [
- repeatAuth := self authenticate: request from: response.
- ].
-
- repeatAuth] whileTrue.
-
- "Flush previous authState.
- XXXX: Fixme. authState must be preserved for pre-authentication of requests."
- self flushAuthState.
-
- "Handle redirect if needed"
- (self allowRedirect and:[response isRedirect]) ifTrue:[
- "Eat up the content of the previous response"
- response content.
- repeatRedirect := self redirect: request from: response.
- ].
- repeatRedirect] whileTrue:[
"When redirecting, remove authentication headers"
request removeHeader: 'Authorization'.
request removeHeader: 'Proxy-Authorization'.
].
+
-
"If the response is not a success, eat up its content"
+ (response isSuccess or: [response isInformational]) ifFalse: [
+ response content].
+
+ ^ response!
- (response isSuccess or:[response isInformational]) ifFalse:[response content].
-
- ^response!