On Friday, February 1, 2002, at 11:53 , Andreas Raab wrote:


> Considering that the average user does run multiple apps and that
> therefore both memory

I was refering to installed real memory.  This *tends* to stay constant 
during program executions.  What were you refering to?

I have seen lots of people try to second-guess a good VM subsystem and 
fail miserably.  (Note:  The NT VM subsystem does not qualify...)

> and disk space

Usually significantly above usable memory as not to be a problem.  As I 
said YMMV.  You are doing multi-gigabyte image-processing with less than 
a couple of gigabytes of disk-space available?  Strange.

> are dynamic resources

The fact that application and OS memory use is highly dynamic is exactly 
the reasoning for letting the OS handle it, not second-guessing it.  Are 
you actually suggesting that Squeak should give a "low memory" warning 
when 'dynamically available' real memory gets low?!?  In that case, you 
would *constantly* get low memory warnings on Mach.

As I said, Mach VM is extremely lazy.  For example, all disk I/O is 
performed by the kernel via memory-mapping.  So if you read() data from 
a file, the kernel will memory-map the file and (vm-)copy that data into 
your address space.

Furthermore, it will not 'free' that memory unless it is acutally needed 
by something more important.  So every file ever read into memory is 
kept in memory unless it is displaced by something else.  This means 
that (real) memory is almost always put to some good use instead of 
being wasted by sitting around empty.  Of course, this also means that 
asking the system for the current amount of free real memory will almost 
always yield a very low number (essentially the head-room the pager 
tries to keep).


> your
> "recommendations" for the user to first see how much disk space and
> memory is available sound just ridiculous.

Actually, the notion that you can (easily) figure out the dynamic state 
of real memory consumption is ridiculous, at least with a sufficiently 
advanced VM subsystem.  The number you get is essentially meaningless, 
as a Mach system in 'steady-state' will keep its real memory close to 
full at all times.  The only times you see significant free memory is 
when you've just quit an app that was busy+large just beforehand.

So how is your auto-grow mechanism going to figure out when to give a 
'low-space' warning?

Furthermore, the VM subsystem will automatically grow additional 
swap-files as needed.  This means that the limit to total non-shared 
memory use is RAM + free-disk.  RAM is fixed, but disk-space happens to 
be variable.  You can of course, be conservative and require backing 
store to be available at all times, but that is restrictive and will 
cause some computations to fail that would otherwise succeed.  Please 
explain why you think that failing earlier, as compared to actually 
succedding, is a good idea.

>  In the end, your scheme risks
> a hard crash if you do any real work in a real-life environment and
> that's most definitely not my understanding of a stable application.

In case you actually have dynamic *memory* requirements that 
significantly outstript available hard-disk space, then there are ways 
of avoiding the issue, not the least of which is the use of the external 
pager interface, which seems to have been reactivated (just checked the 
sources)

> In fact, I'd go sofar and argue that automatic lazy virtual memory
> management is one of the most stupid ideas that I've ever heard of.

I bet the Mach + Darwin developers would be really impressed by your 
opinion.  I can just see them slapping their foreheads, having now 
finally been enlightened, and scurrying to fix their stupid, ridiculous 
VM subsystem.

Not.

>  If the memory subsystem returns you a supposedly valid pointer to a 
> memory
> block you just allocated why should you be unable to use it?

Because (a) other schemes will fail much earlier (b) this scheme works 
better (c) there are various ways of dealing with a failure if you are 
concerned about it happening.

>  If it cannot grant the request it can return Null and have the client 
> deal
> with the problem.

This means both doing more work and failing earlier, in many cases 
failing despite the fact that the request can be fulfilled.

>  With automatic lazy VM you have to be aware that at
> _any_ time you access memory you must be able to recover from a memory
> fault.

This turns out not to be the case.  Only the first time you access 
memory that hasn't been touched yet.  If you're worried about it, use 
calloc() or some other mechanism.  This isn't exactly rocket science.

>  You should also consider that just catching an exception is not
> enough by far - if that happens to you in the midst of trying to write
> to a forwarding block in the middle of full GC you are just screwed.

If you're running out of total disk-space, you're pretty screwed anyhow.

> Requiring the user to "quit and restart" an application in order to get
> some memory back is ... how do I put this ... not exactly close to
> reality.

Try re-reading more slowly.  The behavior *without* the restart is no 
worse than requiring pre-allocated swap.  With the pre-allocated swap, 
you need the disk-space immediately, with the lazy scheme, you just 
might not get it back (though you actually can get it back if you're 
really that concerned about it).

>  What if you want to run a server app? Would you require
> Comanche users to "quit and restart" just because you had a temporary
> large request?!

No, since there is no actual *need* for a restart.

> And your argument that "the system becomes unusable long before" is not
> always true.

Call me when you see a non-contrived counter-example.  I mean, we are 
talking about running out of (total) *disk* space here.

>  I can give you very simple examples that will trigger

Just as any static-typing advocate can give you very simple examples 
that fail with dynamic typing ( -> fail late ) and absolutely 
demonstrate the need for static typing ( -> fail early).  Not impressed.

Marcel