Unsurprisingly, the replies are idea-dense, so I'll summarize below and try to address each idea.

Reviews: 1. Most people can't feel responsible for the whole image, so we need an option to sign up for responsibility for parts of the image, probably in groups. 2. There are few topical experts, and some reviews require experts. 3. Who reviews the experts work? especially a collaboration between a bunch of experts. 4. It sometimes feels silly to review the work of an expert, because of course its good. 5. Maybe we should auto accept their stuff. Testing: 6. Fixes get tested by the person that suffered the bug in the first place. 7. Informal (untagged) feedback gets lost. 8. Maybe tools could help collect feedback?

First of all, I agree with most of these, and think that whatever we do, as long as the harvesting process isnt working well, we should try whatever comes to mind. So while some solutions sound wrong to me, I definitely think we should do experiments, whether I like them or not. I w

About 1 - sounds good to me. How do we kick start this? Avi, would you be willing to take some specific area of Squeak and try to review at least some portion of patches that come through for it? you mentioned maybe having groups that review areas. What would you do to start a group around an area you'd adopt?

2, 3 - note that even in areas where functional dependencies require experts to tell whether the changes are good or not, a non expert is able to review for other things. Andreas, for the example of Ned and you working on morph removal, I think that assuming you'd both looked at all the code, I wouldn't mind if one of you approved it.

Reading 4 and 5, I feel uncomfortable. Looking at the assumptions behind the process for minute, here are mine - A. Everyone is sloppy sometimes, in some way. Inspection by another pair of eyes helps. So everyones work should be looked at by someone else. B. Different people have different standards - this is good, because we can learn from another, if we bother to look at one anothers code critically and discuss our findings. C. What we accept today determines what is posted tommorrow.

A & B are IMO good reasons to have everyones code looked at at least by someone. C is something we need to be aware of when considering optimistic solutions as Marcus is proposing. The worst case scenario of accepting bad fixes isn't that a bug will get in, it is that we will drown in a sea of sloppy fixes, caused by a perception that making sure you got it right is someone elses problem...

So I obviously disagree with 5. I do think we should avoid deadlocking as much as possible - any two people that fully reviewed a patch fulfill the review requirement, even if one or both are coauthors. To signal that they have fully reviewed the code, they should use the [er] tag, and also comment their review. Anyone think that this is still to onerous?

On 6/7/8 - I agree. How do we prompt more bug feedback in a way that can be captured as an [et]? One way that comes to mind is that if people installed bug fixes off the BFAV, because it was included in their image, it might seem more natural for them to also give feedback on it using it.

Daniel

Andreas Raab andreas.raab@gmx.de wrote:

Hi Daniel,

...

Ideas to solve this or alternative explanations would be welcome.

Let me throw in a few thoughts on this issue. Speaking only for myself I find the following issues interesting/relevant in this context:

• External reviews

A key problem with external reviews is that you have to know the area involved very well to judge the wider implications of changes. It is not surprising that typically a few people review certain areas since that's their primary area of expertise. One of the key problems in the current process is that these people need reviews themselves - even though they are experts in the area and an external review is likely to just say "sure it works - what did you expect"? Or even worse, there simply isn't anybody who feels confident to review that code (who exactly even _could_ review Ned's and my joint changes for adding/removing morphs?)

Here's an interesting observation about myself: I tend to _not_ look at code from people I consider "experts" unless I am suspecting something non-obvious or outright questionable. In all other cases my basic inclination is "go ahead - if there's a problem we can fix that too" rather than think ahead and try to foresee what might be affected (which doesn't work anyway).

Personally, there are various people which I would immediately endorse to submit fixes and enhancements in various areas. In other words, I trust those people enough to be willing to live with their changes without having to explicitly review them. And yes, things will break but they'd break with or without a review by someone who doesn't really understand it. If so, what's the point?

• External testing

If you look at the traffic on Squeak-dev, I think you'll find many more responses that just say "thanks! that did it" than "[er, et]" or whatever tags. The problem is that feedback from people using a fix is mostly informal (if at all present). In a perfect world, I would want something that I can flag as a fix for some problem. When some person installs the fix the person should (later) get asked "did this solve your problem?" with a choice of "don't know yet", "yes", "no", "yes but ..." (as in: it fixes the problem but shows another one), and "no but ..." (as in: it doesn't but I have package Foo and Bar loaded which might interfere).

One way to solicit this feedback would be to install in particular [FIX]es in a way that the system _knows_ this is a fix and can ask the user about it after that fix has been in use for a while. Again, here's an example: Someone posts a fix for a problem. I load it and keep going. Some time later (perhaps when I am about to save/quit the image) the system reminds me that I have a bunch of fixes loaded and whether I'd want to comment on them. If I choose to, I get the above mentioned options and this feedback is then automatically sent to the list.

The key point about [et] tags for fixes is that people are individually bitten by the bugs and that loosing feedback from a single person means loosing _all_ of the feedback here. You don't run a [fix] right away unless you're bitten by it.

Cheers,

• Andreas

...

-----Original Message----- From: squeakfoundation-bounces@lists.squeakfoundation.org [mailto:squeakfoundation-bounces@lists.squeakfoundation.org] On Behalf Of Daniel Vainsencher Sent: Thursday, October 16, 2003 5:57 PM To: squeakfoundation@lists.squeakfoundation.org Cc: ken@kencausey.com; Brent Vukmer; Cees de Groot; Bert Freudenberg Subject: [Squeakfoundation]The Harvesting process and the BFAV

Hi guys.

The BFAV has become the technical basis for harvesting progress. In this it has improved the rate of harvesting by removing much of the dirty work that was involved in the old processes. It has improved the efficiency of the approvals many fold.

It has also allowed us to elaborate the process by adding the quality tags, and for the first time having a more or less closed list of patches to be dealt with.

The purpose of the QA tags was to distribute the work of harvesting among more people, by allowing casual participants to help by reviewing or testing code without accepting the responsibility of the formal role of harvester.

As the efficiency of approvals went up, and we have an in-squeak list of posts and their full status, it has become much more visible that our bottleneck is not approvals. It is very clearly in the reviewing/testing stage. The reasons are very clear - rather few people help with the reviews/testing, and that quite sporadically.

This bottleneck is a result of our current structure, and won't go away by itself. It causes a situation where most patches meant for the squeak image are ignored. I think this is not likely to last - either we balance this by removing the bottleneck, or contributors of patches will balance it by posting less, an outcome I find regrettable.

In fact, I think it likely that people are already deciding not to post fixes they have available because they're likely to be ignored. Therefore, I think that if we can remove the bottleneck, and reach a state where patches submitted are all processed in reasonable time, we will get far more patches, and be able to make much better progress than we are now.

I think the following steps are in order -

1. We want to lower the barriers to casual participation. So,

the BFAV, as a prerequisite for participating, should never be absent for anyone. It should be included in the base image. 2. We should improve and simplify the tools to help reviewers/testers do their job.

• ClassBoxes, when they're ready, could become a big aid to

testing, by allowing patches to be installed temporarily and then completely removed.

• The BFAV should be improved, especially for casual users. Brent, I

recommend to open its development so more people can help with this. The Swiki page is already pretty good, just point to it from the help in the BFAV. Also it might be good that its future development be carried on using Monticello, to make it easier to create and integrate improvements. 3. The current flood of BFAV traffic makes people automatically ignore the harvesting process. Therefore it MUST be selectively moved away from squeak-dev. Cees, or whoever has administrative rights on lists.sqf.org, can you please create a new mailing list called "squeak-harvest"? This fits Berts existing filters. Having this list would allow some selectivity without losing information. 4. When all is said and done, many people currently won't participate no matter how low the barriers are made, because of different priorities. Some of this is obviously reasonable, people need to pay the rent and play with whats fun for them, of course. But some of this is IMO not reasonable, and comes from a misperception. I think the NewLook code showcases this very clearly. The voices for its inclusion are very loud, but nobody bothers to review it, therefore, it isn't included. So I guess that the people doing the requesting somehow fail to perceive that the condition for inclusion is within their grasp. Of course, I could just be wrong about this. Ideas to solve this or alternative explanations would be welcome.

Daniel _______________________________________________ Squeakfoundation mailing list Squeakfoundation@lists.squeakfoundation.org http://lists.squeakfoundation.org/listinfo/squeakfoundation

---

Squeakfoundation mailing list Squeakfoundation@lists.squeakfoundation.org http://lists.squeakfoundation.org/listinfo/squeakfoundation