--On Sunday, August 26, 2001 3:40 PM +0100 John Hinsley jhinsley@telinco.co.uk wrote: [snip]

That's a huge leap forward.

I'm begining to see just how very difficult this is! Any ideas about this one? Weblint shows it as perfect, but Scamper chokes (even with your new .cs).

[snip HTML]

(amazing what old stuff you find lying about on your hard drive!)

[snip]

In general, for this sort of thing, I recommend using HTML-Tidy to preclean the input into XHTML. XHTML is *way* more regular. Also, HTML-Tidy does a pretty good job of handling random HTML pretty much the way the major broswer(s? is there is a second major browser? :)) does.

I've been poking at HTML-Tidy with a thought of a Squeak port (there's a Java port), but haven't had any time at all.

Note that Georg Heeg's public wiki (which I found once and never again :)) has a T-Gen based HTML parser which prefers HTML-Tidy sanitized input. One nice thing is that they have a full array of classes for representing HTML 4.0. It's for VisualWorks, but it can't be that hard to move over.

Cheers, Bijan Parsia.

John Hinsley wrote:

Karl Ramberg wrote:

...

I'm still torturing my self, but I begin to see the light :-) -I solved the nesting of the tables! -links within tables work These were the really hard parts to figure out.

That's a huge leap forward.

Thanks:-)

I'm begining to see just how very difficult this is! Any ideas about this one? Weblint shows it as perfect, but Scamper chokes (even with your new .cs).

Eeeeeeeeek, buggy html ! The Squeak html parser is not up to the task as far as I can see. It's hard to understand how it works too, so I'm not going there today;-)

(amazing what old stuff you find lying about on your hard drive!)

:-)

Karl

On Sun, 26 Aug 2001 19:05:27 +0200, Karl Ramberg karl.ramberg@chello.se wrote:

Eeeeeeeeek, buggy html ! The Squeak html parser is not up to the task as far as I can see. It's hard to understand how it works too, so I'm not going there today;-)

As I said in one of my earlier posts, probably more than half of the time I spent working on the Interval web browser was spent making it deal with bad HTML.

At the time, I decided to make something that would take the parse tree, and try and correct any mistakes or problems it had. It was very annoying, given that it is a job that never seemed to be done...

Later, Jon

-------------------------------------------------------------- Jon Hylands Jon@huv.com http://www.huv.com/jon

Project: Micro Seeker (Micro Autonomous Underwater Vehicle) http://www.huv.com

Ned,

And the example was actually valid HTML 3.2. It's just that you have to

add

logic for omitted end tags like </td> and </tr>.

I believe that the push is for all tags to be balance, as with XML. But, yes, one can add logic for a set of tags to close the current DOM child and open a new one. However, I would suggest that he focus the core of what he is doing on making it work with properly balanced tags, and let the fixups occur earlier in the flow.

--- Noel

"Noel J. Bergman" wrote:

John Hinsley's table example is very ill-formed.

No it's not! It's -- as Ned points out -- absolutely valid and weblint verified. If you want to argue that pre pseudo XML HTML is ill-formed, you'll have one hell of a lot of valid sites to re-write! (Then you can start on the genuinely ill-formed ones ;-) ).

Tidy is a nice tool. My main objections to it are that it balances tags where they really don't need to be balanced -- unecessarily increasing the length of documents is _always_ a bad thing -- and that it inserts its own meta tag (see above, plus it's rather rude).

What I like about HTML is that I can read a page written to the earliest standards in the latest browsers. (Scamper excepted!) If someone sends me a Word 2000 .doc as an email attachment I can always reply and tell them to send it in another format (if I'm feeling particularly awkward, I'll ask for it as dvi). Were my browser to refuse to read HTML to earlier standards, I'd be buggered. (Seems to me that this may be exactly what Microsoft want.)

(Incidentally, I know of at least one post-grad course where people are still taught to hand craft 3.2 HTML with vi -- I was accused of cheating because I found a copy of gvim -- built against Motif -- yuk! -- on one of the servers and thus had the advantage of coloured syntax highlighting.)

But as I'm not in a position to do any work on it, I wouldn't want to enter into any argument about what course Scamper's development should follow. Follow your heart, Karl: by doing the work, you've earned that right!

Cheers

John

John Hinsley wrote:

"Noel J. Bergman" wrote:

...

John Hinsley's table example is very ill-formed.

Tidy is a nice tool. My main objections to it are that it balances tags where they really don't need to be balanced -- unecessarily increasing the length of documents is _always_ a bad thing -- and that it inserts its own meta tag (see above, plus it's rather rude).

The HtmlParser in the image is basically one method that steps trough the html and it is looking for balanced tags. A update for it to deal with html standards would be really nice, but I'm not in the mood tonight :-)

What I like about HTML is that I can read a page written to the earliest standards in the latest browsers. (Scamper excepted!) If someone sends me a Word 2000 .doc as an email attachment I can always reply and tell them to send it in another format (if I'm feeling particularly awkward, I'll ask for it as dvi). Were my browser to refuse to read HTML to earlier standards, I'd be buggered. (Seems to me that this may be exactly what Microsoft want.)

I also like this about html. It's a very simple and quite powerful way to distribute documents. But I agree with Noel here, you can never rely on the html to be correct so pushing it trough a cleaner is not the ideal solution, but seems necessary.

But as I'm not in a position to do any work on it, I wouldn't want to enter into any argument about what course Scamper's development should follow. Follow your heart, Karl: by doing the work, you've earned that right!

First I will pursuit the task of laying out the tables that the current parser digests. I have some problems with how to get them to scale properly when no size attributes are given. Suggestions are welcome :-)

Karl

Bijan Parsia wrote:

--On Monday, August 27, 2001 1:48 AM +0100 John Hinsley jhinsley@telinco.co.uk wrote:

[snip]

...

Tidy is a nice tool. My main objections to it are that it balances tags where they really don't need to be balanced -- unecessarily increasing the length of documents is _always_ a bad thing

No, it's not, especially if it makes parsing easier.

A stronger objection to Tidy is having a two pass rendering -- for many machines this would be unbearably slow, espeically as Scamper (to my knowledge) doesn't (yet) cache rendered pages.

I did a quick test with the source code from minnows quote of the day page. Parsing the page takes 470 milliseconds and formatting the page with the table support takes 2200 milliseconds. This is on a old 150 mz mac. Thats not too bad:-)

Karl

Karl Ramberg wrote:

Well, know I tracked down the culpit of the problem and that was me... I had changed the mayContain in HtmlTableDataItem to mayContain: anEntity ^ true.

Check out the attached change set to see if it helps :-)

Wow! That does it. (But you already knew this! ;-) )

Of course, the other improvement is that the backgrounds to the table data are no longer green, but the "proper" colour, that is, the background as declared in the HTML. Thinking back to where Scamper was at less than a fortnight ago, this is a huge leap for Squeakkind!

Indeed, although Scamper still doesn't display everything quite as it should (it does some strange stuff with alignments and has problems with some of the truly bad HTML written by Word and StarOffice) it's now very close to being a perfectly useable browser.

Well done Karl!

If you're feeling in a particularly masochistic mood, you might like to take a peek at the attached!

Thanks again

Cheers

John

John Hinsley wrote:

If you're feeling in a particularly masochistic mood, you might like to take a peek at the attached!

Yup, this is were I'm at at the moment. There is a subtle diffrence in how tables are defined used in html and Squeak. I'm still not sure whats the best approach yet. What I do now is add a TableMorph, then add a row morph, and then a cell morph. But I'm starting to think that the row morph is unnessesary. But then I don't know how to wrap the cells at the right place...

Karl

Lex Spoon wrote:

Oh man, this thread is so painful to read. It's almost enough to make me want to document this stuff. :) Although actually, some of it is documented on the Scamper swiki page (or, some page nearby to it. I don't have Internet access to check the exact location right now.)

Well, reading code and adding to is is a sweet pain:-)

Close tags are *not* required. If they were then lists wouldn't work, because people hardly ever put in the </li>'s. In fact, Scamper copes not only with omitted close tags, but with lots of erroneous HTML, as well. This is why it uses a hand-written parser instead of a much more elegant SGML parser.

I'm really impressed of the parser. It's basically one method, and it handles dreadfull html really good.

(A nice thing about XML is a social one: it's so verbose and obscure that nobody wants to write it by hand, and so it's socially acceptible for parser's to just dump even slightly erroneous input on the floor.)

The mayContain: idea is crucial for both of these, and the parsing problem people saw is surely because one or more of the various HtmlTable class's has an incorrect mayContain: method. It's worth spending a few minutes getting these #mayContain: methods to something reasonable, instead of just returning true all the time -- it helps with making reasonable parses of bad HTML, and it helps when processing the HTML after it's parsed, because you know, e.g., that TABLE's only contain TR's.

My problem ocurred when TD not could contain TABLES, therefor I changed the mayContain to return true and got into all kind of truble. I fixed this in the third version I posted and it works the way it should now.

Okay, those little bits aside, here is a skeleton design for adding tables. I'll leave out the details of building the table morph itself -- frankly, I don't even know exactly how! :)

The core idea is to make outputStream be a stack, ie "outputStreams" instead of "outputStream". Add an #outputStream method that returns the top element of outputStreams, update all users of the outputStream ivar to use the #outputStream method, and then add some methods like:

    #startNewOutputStream
    #endCurrentOutputStream

(Here would be a good time to run regression tests. Ah well, too bad there aren't any.)

Now, to format a table, add a addToFormatter: method to HtmlTable like this:

addToFormatter: formatter | tm | tm := "insert your code to make a table morph". self subEntities do: [ :rowEntity | tm startNewRow. rowEntity subEntities do: [ :cell | formatter startNewOutputStream. cell addToFormatter: formatter. tm addCell: formatter formattedText. formatted endCurrentOutputStream ] ]. formatter addMorph: tm

Note, critically, that #formattedText should return the text of the *current* output stream, not of of the whole document. This will probably happen automatically, but be aware of this as you read the above code.

Here is what I have done: I add tables as new morph to formatter. I add rows as submorphs to the last table in the formatter. (The layout is done with the TableLayout stuff added in 3.1alpha) And at last the cell is added with this code:

addToFormatter: formatter | data nuDocument nuFormatter m | data _ TableDataMorph new. "adds the new cell and sets up it's layout" data color: Color transparent. data borderColor: Color black. data borderWidth: 1. data layoutPolicy: TableLayout new. data listDirection: #leftToRight. data wrapCentering: #topLeft. data hResizing: #spaceFill. data vResizing: #spaceFill. data layoutInset: 0. formatter addTableDataMorph: data." add cell to last table's last row!" nuFormatter _ HtmlFormatter preferredFormatterClass new. "set up a new formatter" nuFormatter browser: formatter browser. nuFormatter baseUrl: formatter baseUrl. 1 to: (contents size) do:[:i| nuDocument _ contents at: i. nuDocument addToFormatter: nuFormatter. ]. " feed the formatter with whatever is present" data addMorphBack: nuFormatter textMorph. "add the textMorph made from the new formatter to the cell" ((m _ nuFormatter incompleteMorphs size) > 0) ifTrue:[ 1 to: m do: [:i| formatter incompleteMorphs add: (nuFormatter incompleteMorphs at:i)]]. "make sure images get updatet"

And that's it. Now you just have to figure out how to make TableMorph's at all!

My problem now is that the layout policy in squeak does not handle row spans and column spans the way html tables does, as far as I know ;-) Any way, I think I will change stuff around a little. Hopefully there is no need for a seperate row morph, and by removing that there will be a way to get the cells in a table layout to wrap and deal with row spans and colspans. Feedback on this is wanted !

By the way, here are two other tricks people will be interested in, once you start playing with tables. This stuff I have not looked into. My apprach is the Morphic Layout Way:-) First, you can get tables to resize horizontally by adding some sort of widthProportion instance variable to TextAnchor (TextAnchor is the subclass of TextAttribute that Text uses to deal with embedded morphs). Then, TextAnchor>>emphasizeScanner can resize the morph before doing its normal thing, if this instance variable has been set. Bolot and I put together something like this once upon a time, but it got lost. I leave figuring out the width of a CharacterScanner as an excercise for those who have a lot of time.

Second, you can add a TextAlignment attribute that allows centered text, but this is a tad more work. You must make all uses of "textStyle alignment" in CharacterScaner+subclasses use the "alignment" instance variable, instead. The rest is really easy: add an #allignment: method in TextAlignment that sets the alignment, and make TextAlignment>>emphasizeScanner: set the alignment on the supplied formatter.

Cheers! Like Je77, I'm really excited to see people playing with this -- tables would make Scamper much nicer to use!

By the way, frames aren't too far away, once we have tables -- one could use the same table morph that tables use, but with some sort of "DownloadingHtmlMorph" in the cells. DownloadingHtmlMorph will look like DownloadingImageMorph, except that it downloads and displays HTML instead of images.

There is allready a WebPageMorph that is perfect for the task! Once the row span col span stuff is worked out I'll look into it.

My apologies for not reading any of the specific table packages people have posted. The above fixes all of the problems people have mentioned having, though, and so unless everyone is burned out now I'll wait until someone has a more complete solution.

You as everybody else are welcome to jump in at any time:-)

Karl

Lex Spoon wrote:

...

My problem ocurred when TD not could contain TABLES, therefor I changed the mayContain to return true and got into all kind of truble. I fixed this in the third version I posted and it works the way it should now.

Ah, good example. It seems that HtmlTable's are currently neither "block entities" nor "textual entities", and thus are trying to be at the top-level of the document. Adding isBlockEntity to return true would make sense, and it would seem to fix this particular problem as well as others.

I made it return true to isTable.

...

  nuFormatter _ HtmlFormatter preferredFormatterClass new. "set up a new  formatter"
  nuFormatter browser: formatter browser.
  nuFormatter baseUrl: formatter baseUrl.
  1 to: (contents size) do:[:i| nuDocument _ contents at: i.
                  nuDocument addToFormatter: nuFormatter. ]. " feed the formatter with

whatever is present"

Ahh, you create a new formatter for each cell. That should work okay, though it doesn't inherit any settings from the current formatter except for ones you copy explicitly.

And its kind of slow... but it gives the advantage of using the morphic table layout. I also fixed somestuff with TextMorph so embedded morphs show up, hopefully.

But, it certainly sounds good enough as is to be very useful! Yay Karl!

It works now, exept the darn row and coll span stuff... and the layout does not pay attention to attributes like borders and spacing etc. yet.

Oh, an other thing that don't work is tiled pictures, like the top and bottom line in the table on the Squeak Swiki.

Karl