A new version of File-Base was added to project Rio: http://www.squeaksource.com/Rio/File-Base-kph.3.mcz
==================== Summary ====================
Name: File-Base-kph.3 Author: kph Time: 25 November 2008, 5:13:56 am UUID: 86b85984-2283-48b6-8229-00a4dade4f75 Ancestors: File-Base-kph.2
+ multithreaded ftp now working up/down - archive is broken
=============== Diff against File-Base-kph.2 ===============
Item was changed: FileRemoteExecutive subclass: #FileFtpExecutive + instanceVariableNames: 'home isKeepAlive rw' - instanceVariableNames: 'home isKeepAlive rw ftpClient' classVariableNames: '' poolDictionaries: '' category: 'File-Base'!
Item was changed: ----- Method: Directory>>addAll: (in category 'directory/container actions') ----- addAll: someFD + self addAll: someFD relativeTo: nil! - self addAll: someFD fromBase: nil!
Item was added: + ----- Method: Directory>>addAll:relativeTo: (in category 'directory/container actions') ----- + addAll: someFD relativeTo: aBaseDir + + "someFd must all have the same executor" + + "we could perform a sort here to handle the other case" + + someFD isEmpty ifTrue: [ ^ self ]. + + someFD first executive addAll: someFD relativeTo: aBaseDir toDir: self.!
Item was added: + ----- Method: FileRemoteExecutive>>initialize (in category 'as yet unclassified') ----- + initialize + + client := nil.!
Item was added: + ----- Method: FileFtpExecutive>>toDirLocal:addAllRemote:relativeTo: (in category 'local/remote file copy') ----- + toDirLocal: aDir addAllRemote: someFD relativeTo: aBaseDir + + self toDirRemote: aDir addAllLocal: someFD relativeTo: aBaseDir + copySelector: #copyRemoteFile:toLocalFile:!
Item was added: + ----- Method: FileFtpExecutive>>toDir:addAllLocal:relativeTo: (in category 'local/remote file copy') ----- + toDir: aDir addAllLocal: someFD relativeTo: aBaseDir + + ^ self toDirRemote: aDir addAllLocal: someFD relativeTo: aBaseDir!
Item was changed: ----- Method: FileFtpExecutive>>copyRemoteFile:toLocalFile: (in category 'external ftp') ----- copyRemoteFile: aFile toLocalFile: bFile
self class OSProcessOrNil ifNil: [ ^ aFile copyTo: bFile ]. self class OSProcessOrNil waitForCommand: 'ftp -o "', bFile asVmPathName, '" "' , url asString , aFile, '"'!
Item was changed: ----- Method: FileFtpExecutive>>copyRemoteFile:toRemoteFile: (in category 'external ftp') ----- copyRemoteFile: aFile toRemoteFile: bFile
+ "we copy the executives to enable multi threaded copying and because we cant read and write with the same executive" + + ^ (aFile executive: aFile executive copy initialize) copyTo: (bFile executive: bFile executive copy initialize)! - ^ aFile copyTo: bFile!
Item was added: + ----- Method: FileFtpExecutive>>toDir:addAllRemote:relativeTo: (in category 'local/remote file copy') ----- + toDir: aDir addAllRemote: someFD relativeTo: aBaseDir + + ^ self toDirRemote: aDir addAllRemote: someFD relativeTo: aBaseDir!
Item was changed: ----- Method: FileFtpExecutive>>initialize (in category 'initialization') ----- initialize
+ super initialize. isKeepAlive := false.!
Item was added: + ----- Method: FileFtpExecutive>>toDirRemote:addAllLocal:relativeTo: (in category 'local/remote file copy') ----- + toDirRemote: aDir addAllLocal: someFD relativeTo: aBaseDir + + self toDirRemote: aDir addAllLocal: someFD relativeTo: aBaseDir + copySelector: #copyLocalFile:toRemoteFile:!
Item was added: + ----- Method: FileFtpExecutive>>toDirRemote:addAllLocal:relativeTo:copySelector: (in category 'local/remote file copy') ----- + toDirRemote: aDir addAllLocal: someFD relativeTo: aBaseDir copySelector: aSelector + + "here we do two passes, first we ensure that all of the needed directories exist. + for this we use our ususal ftp client. + + for the file transfers themselves we run n threads feeding from a shared queue + + " + + | queue map done delay | + + self ftpDo: [ :ftp | + + map := self toDir: aDir mkpathAll: someFD relativeTo: aBaseDir. + + ]. + + delay := Delay forMilliseconds: 100. + queue := SharedQueue2 new setItems: map copy. + done := SharedQueue2 new. + + self class prefFtpParallel timesRepeat: [ + [ + [ queue isEmpty ] whileFalse: [ + queue nextOrNil ifNotNilDo: [ :ea | + [ self perform: aSelector withArguments: ea ] ensure: [ done nextPut: ea ]. + ] + ] + ] fork. + ]. + + [ done size < map size ] whileTrue: [ delay wait ]. + + ^ map!
Item was changed: ----- Method: Directory>>add: (in category 'directory/container actions') ----- add: aFD + ^ self addAll: (Array with: aFD) ! - ^ self add: aFD fromBase: nil - !
Item was added: + ----- Method: FileFtpExecutive>>toDirRemote:addAllRemote:relativeTo: (in category 'local/remote file copy') ----- + toDirRemote: aDir addAllRemote: someFD relativeTo: aBaseDir + + self toDirRemote: aDir addAllLocal: someFD relativeTo: aBaseDir + copySelector: #copyRemoteFile:toRemoteFile:!
Item was added: + ----- Method: FileFtpExecutive>>addAll:relativeTo:toDir: (in category 'local/remote file copy') ----- + addAll: someFD relativeTo: aBaseDir toDir: aDir + + ^ aDir executive toDir: aDir addAllRemote: someFD relativeTo: aBaseDir.!
Item was changed: ----- Method: Directory>>addTree: (in category 'directory/container actions') ----- addTree: aDir - - | base newAddition | self validateIsContainer. aDir validateIsDirectory. + + ^ self addAll: (aDir all entries addFirst: aDir; yourself) relativeTo: aDir parent. - - newAddition := self add: aDir fromBase: (base := aDir parent). - - self addAll: (aDir all entries) fromBase: base. + ! - ^ newAddition - !
Item was removed: - ----- Method: FileFtpExecutive>>dir:add:fromBase: (in category 'basic') ----- - dir: aDir add: aFileOrDir fromBase: aBaseDirectory - self ftpDo: [ :ftp | super dir: aDir add: aFileOrDir fromBase: aBaseDirectory ] !
Item was removed: - ----- Method: Directory>>addAll:fromBase: (in category 'directory/container actions') ----- - addAll: someFD fromBase: aDirectory - - executive dir: self addAll: someFD fromBase: aDirectory!
Item was removed: - ----- Method: FileFtpExecutive>>dir:addAll:fromBase: (in category 'basic') ----- - dir: aDir addAll: someFD fromBase: aBaseDir - - "here we do two passes, first we ensure that all of the needed directories exist. - for this we use our ususal ftp client. - - for the file transfers themselves we run n threads feeding from a shared queue - - " - - | queue items done | - - self ftpDo: [ :ftp | - - self class OSProcessOrNil ifNil: [^ super dir: aDir addAll: someFD fromBase: aBaseDir ]. - - items := someFD collect: [ :aFileOrDir | (self dir: aDir mkpath: aFileOrDir fromBase: aBaseDir) -> aFileOrDir ]. - items := items select: [ :ea | ea value isFile ]. - ]. - - queue := SharedQueue2 new setItems: items. - done := SharedQueue2 new. - - self class prefFtpParallel timesRepeat: [ - [ - [ queue isEmpty ] whileFalse: [ - queue nextOrNil ifNotNilDo: [ :ea | - self dir: ea key addFile: ea value. - done nextPut: 1 - ] - ] - ] fork. - ]. - - [ done size < items size ] whileTrue: [ Delay forMilliseconds: 100 ]. - !
Item was removed: - ----- Method: Directory>>add:fromBase: (in category 'directory/container actions') ----- - add: aFileOrDir fromBase: aBaseDirectory - - "I am a directory, add the file or create directory using aBaseDirectory - as the base reference." - - self validateIsDirectory. - - executive dir: self add: aFileOrDir fromBase: aBaseDirectory - - - !
Item was removed: - ----- Method: FileFtpExecutive>>copyFile:toLocalFile: (in category 'external ftp') ----- - copyFile: aFile toLocalFile: bFile - - ^ self copyRemoteFile: aFile toLocalFile: bFile!
Item was removed: - ----- Method: FileFtpExecutive>>copyFile:toRemoteFile: (in category 'external ftp') ----- - copyFile: aFile toRemoteFile: aDir - - ^ self copyRemoteFile: aFile toRemoteFile: aDir!
Item was removed: - ----- Method: FileArchive>>add: (in category 'public') ----- - add: aFD - - self validateIsContainer. - - executive dir: self add: aFD !
Item was removed: - ----- Method: FileArchive>>add:fromBase: (in category 'public') ----- - add: aFD fromBase: aBaseDirectory - - "I am an archive, add the file or create directory using aBaseDirectory - as the base reference." - - self validateIsContainer. - - executive dir: self add: aFD fromBase: aBaseDirectory!
packages@lists.squeakfoundation.org