<div dir="ltr"><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div>Hi Phil,<br><br></div>that's probably right: there is a lack of smoke tests.<br></div><br></div>There are several hurdles before we reach today's state of the art wrt continuous delivery and regression testing<br><br></div>- 1) the artefacts must be built, or we won't even have a chance to run tests<br></div> we can observe that they have been broken too many times by all sort of problems, and green status is an exception in an ocean of red<br> problems encounterd so far are including<br></div><div> * work in progress in core VM or plugins</div><div> * wrong configuration of pharo target directories or credentials<br> (this was the case most of the 2017 year but is fortunately fixed now)<div> * staled or intermitent links (url)<br></div><div> for example, the build is loading stuff from the network (like cygwin updates)<br></div><div> that sometimes fail<br></div></div> * failure to build a library due to some tool changes at appveyor/travis</div><div><br></div><div>Introduction of new bugs could be prevented if feedback was correct (no false alarm).<br></div><div>But it's not really the case until now (lot of parasites).<br><br></div>- 2) we run after many hares, that is a combination of<br></div> v3 stack spur, i386 x86_64 ARM, linux Windows MacOS, sista lowcode, Squeak Pharo Newspeak<br></div><div> I certainly forgot threaded FFI in above list, plus the register efficient JIT variants...<br></div><div><br></div>Again, breaking a single of these configurations lead to RED status.<br></div>Not all these configurations are at the same level<br></div>- of importance (less user, not used in production, ...)<br></div>- of maturity (in progress, experimental, or in production)<br></div>So we must find a way to prioritize and focus on production artifacts...<br><br></div>-3) we need stable image side for running smoke tests<br></div>But we need some image side changes for some new features, preventing to run older versions.<br></div>Squeak and Pharo still have randomly failing tests (like Network dependent, etc...).<br></div><div>Someone has to do the work (or pay it)...<br></div><div><br></div><div>-4) build status feedback is very sloowww<br><div> * as said above, we build too many configurations<br></div><div> * Pharo has introduced a lot of dependencies on external libraries<br></div><div> this leads to either long build times, or the use of caches that delay detection of new failures<br></div><br></div>We all know that dev branches (feature branches) help a lot for some of the above problems.<br></div>But we have these additional hurdles:<br></div>- feature branches works well when cycles are short<br></div> but core VM cycles are not short (3 to 6 months or more for introducing new GC, 64 bits, minimal SISTA, ...)<br></div><div> a lot of the changes required for SISTA, 64bits and JIT variants are competing, and parallel branches would create conflicts and would not work without regular sync.<br></div> that explains why all the branches are gathered into a giant and complex one today...<br></div>- versionning generated code is a recipe for creating unsolvable conflicts (unmergeable)<br></div><div> it's still possible to generate code for a plugin (if non concurrent)<br></div> but this prevents working in parallel branches as soon as the core generation is changed in VMMaker</div><br>In recent posts, I saw billiant young people under-estimating a bit the work involved and the complexity of the task.<br></div>Fabio has done a tremendous work to restore the green status, and the help of Esteban has been decisive with this respect.<br></div>We will never thank them enough for that.<div><br></div><div>But maybe current state is at the limit of sustainability.<br>And maybe it's time to drop some drag.</div><br><div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">2018-03-30 22:35 GMT+02:00 Phil B <span dir="ltr"><<a href="mailto:pbpublist@gmail.com" target="_blank">pbpublist@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <br><div dir="auto">While I've been enjoying the fantastic performance improvements we've seen from Cog onward, one thing I've been less excited about are some of the stability/functionality issues I've been running into. They are not numerous (maybe 1/2 dozen or so major ones in the last 5 years) but they are getting quite tedious to isolate and replicate. Recent examples that come to mind include the 64-bit primHighResClock truncation and 'could not grow remembered set' issues (My current joy is a case where I have an #ifTrue: block that doesn't get executed unless I convert it to an #ifTrue:ifFalse: with a noop for the ifFalse:.. I'll provide a reproducible test case as soon as I'm able. The specific issue isn't the issue, but rather that I keep hitting things like this that seem rather fundamental yet edge-casey at the same time)<div dir="auto"><br></div><div dir="auto">I don't expect perfection as a phenomenal amount of progress is being made by a small group of people but I am beginning to wonder if the existing unit tests are sufficient to adequately exercise the VM? I.e. so that the VM developers are aware that a recent change may have broken something or are the existing tests mainly oriented towards image and bytecode VM development? Just some food for thought and wanted to see if it's just me having these sorts of issues...</div><div dir="auto"><br></div><div dir="auto">Thanks,</div><div dir="auto">Phil</div></div>
<br></blockquote></div><br></div>