Marcel,

Let's get back to square one here. First of all, early versions of the
Windows VM used _exactly_ the scheme you were describing. So I've been
there and done that. The reason why I don't like it is that
a) without backing store you can never be sure that the memory is really
available
b) with backing store you waste huge amount of resources
c) you are unable to give memory back to the OS.
All of the above are real problems; I have encountered every single
problem I've been mentioning (in practice a little more complicated than
the one I cited but in spirit they were very similar). Or why else would
I have bothered actually adding the growth/shrink mechanism in the first
place?! ;-)

BTW, for me this issue is not about "what OS is better or worse" or
"which memory subsystem is better". I don't give a damn about that; in
the end all of of them are nothing but hardware abstractions for Squeak.
This issue is about stability of Squeak and while it may be fine that a
generic Unix app just coredumps if it has no more memory I don't quite
think that just because some user typed "Array new: SmallInteger maxVal"
Squeak should do the same. I'm sure you'll agree on that.

Cheers,
  - Andreas

> -----Ursprüngliche Nachricht-----
> Von: squeak-dev-admin at lists.squeakfoundation.org 
> [mailto:squeak-dev-admin at lists.squeakfoundation.org] Im 
> Auftrag von Marcel Weiher
> Gesendet: Samstag, 2. Februar 2002 01:01
> An: squeak-dev at lists.squeakfoundation.org
> Betreff: Re: AW: AW: Dynamic system memory use
> 
> 
> 
> On Friday, February 1, 2002, at 11:53 , Andreas Raab wrote:
> 
> 
> > Considering that the average user does run multiple apps and that
> > therefore both memory
> 
> I was refering to installed real memory.  This *tends* to 
> stay constant 
> during program executions.  What were you refering to?
> 
> I have seen lots of people try to second-guess a good VM 
> subsystem and 
> fail miserably.  (Note:  The NT VM subsystem does not qualify...)
> 
> > and disk space
> 
> Usually significantly above usable memory as not to be a 
> problem.  As I 
> said YMMV.  You are doing multi-gigabyte image-processing 
> with less than 
> a couple of gigabytes of disk-space available?  Strange.
> 
> > are dynamic resources
> 
> The fact that application and OS memory use is highly dynamic 
> is exactly 
> the reasoning for letting the OS handle it, not 
> second-guessing it.  Are 
> you actually suggesting that Squeak should give a "low 
> memory" warning 
> when 'dynamically available' real memory gets low?!?  In that 
> case, you 
> would *constantly* get low memory warnings on Mach.
> 
> As I said, Mach VM is extremely lazy.  For example, all disk I/O is 
> performed by the kernel via memory-mapping.  So if you read() 
> data from 
> a file, the kernel will memory-map the file and (vm-)copy 
> that data into 
> your address space.
> 
> Furthermore, it will not 'free' that memory unless it is 
> acutally needed 
> by something more important.  So every file ever read into memory is 
> kept in memory unless it is displaced by something else.  This means 
> that (real) memory is almost always put to some good use instead of 
> being wasted by sitting around empty.  Of course, this also 
> means that 
> asking the system for the current amount of free real memory 
> will almost 
> always yield a very low number (essentially the head-room the pager 
> tries to keep).
> 
> 
> > your
> > "recommendations" for the user to first see how much disk space and
> > memory is available sound just ridiculous.
> 
> Actually, the notion that you can (easily) figure out the 
> dynamic state 
> of real memory consumption is ridiculous, at least with a 
> sufficiently 
> advanced VM subsystem.  The number you get is essentially 
> meaningless, 
> as a Mach system in 'steady-state' will keep its real memory close to 
> full at all times.  The only times you see significant free memory is 
> when you've just quit an app that was busy+large just beforehand.
> 
> So how is your auto-grow mechanism going to figure out when to give a 
> 'low-space' warning?
> 
> Furthermore, the VM subsystem will automatically grow additional 
> swap-files as needed.  This means that the limit to total non-shared 
> memory use is RAM + free-disk.  RAM is fixed, but disk-space 
> happens to 
> be variable.  You can of course, be conservative and require backing 
> store to be available at all times, but that is restrictive and will 
> cause some computations to fail that would otherwise succeed.  Please 
> explain why you think that failing earlier, as compared to actually 
> succedding, is a good idea.
> 
> >  In the end, your scheme risks
> > a hard crash if you do any real work in a real-life environment and
> > that's most definitely not my understanding of a stable application.
> 
> In case you actually have dynamic *memory* requirements that 
> significantly outstript available hard-disk space, then there 
> are ways 
> of avoiding the issue, not the least of which is the use of 
> the external 
> pager interface, which seems to have been reactivated (just 
> checked the 
> sources)
> 
> > In fact, I'd go sofar and argue that automatic lazy virtual memory
> > management is one of the most stupid ideas that I've ever heard of.
> 
> I bet the Mach + Darwin developers would be really impressed by your 
> opinion.  I can just see them slapping their foreheads, having now 
> finally been enlightened, and scurrying to fix their stupid, 
> ridiculous 
> VM subsystem.
> 
> Not.
> 
> >  If the memory subsystem returns you a supposedly valid 
> pointer to a 
> > memory
> > block you just allocated why should you be unable to use it?
> 
> Because (a) other schemes will fail much earlier (b) this 
> scheme works 
> better (c) there are various ways of dealing with a failure 
> if you are 
> concerned about it happening.
> 
> >  If it cannot grant the request it can return Null and have 
> the client 
> > deal
> > with the problem.
> 
> This means both doing more work and failing earlier, in many cases 
> failing despite the fact that the request can be fulfilled.
> 
> >  With automatic lazy VM you have to be aware that at
> > _any_ time you access memory you must be able to recover 
> from a memory
> > fault.
> 
> This turns out not to be the case.  Only the first time you access 
> memory that hasn't been touched yet.  If you're worried about it, use 
> calloc() or some other mechanism.  This isn't exactly rocket science.
> 
> >  You should also consider that just catching an exception is not
> > enough by far - if that happens to you in the midst of 
> trying to write
> > to a forwarding block in the middle of full GC you are just screwed.
> 
> If you're running out of total disk-space, you're pretty 
> screwed anyhow.
> 
> > Requiring the user to "quit and restart" an application in 
> order to get
> > some memory back is ... how do I put this ... not exactly close to
> > reality.
> 
> Try re-reading more slowly.  The behavior *without* the restart is no 
> worse than requiring pre-allocated swap.  With the 
> pre-allocated swap, 
> you need the disk-space immediately, with the lazy scheme, you just 
> might not get it back (though you actually can get it back if you're 
> really that concerned about it).
> 
> >  What if you want to run a server app? Would you require
> > Comanche users to "quit and restart" just because you had a 
> temporary
> > large request?!
> 
> No, since there is no actual *need* for a restart.
> 
> > And your argument that "the system becomes unusable long 
> before" is not
> > always true.
> 
> Call me when you see a non-contrived counter-example.  I mean, we are 
> talking about running out of (total) *disk* space here.
> 
> >  I can give you very simple examples that will trigger
> 
> Just as any static-typing advocate can give you very simple examples 
> that fail with dynamic typing ( -> fail late ) and absolutely 
> demonstrate the need for static typing ( -> fail early).  Not 
> impressed.
> 
> Marcel
> 
> 
>