[Squeakfoundation]The Harvesting process and the BFAV

[Daniel Vainsencher]

Unsurprisingly, the replies are idea-dense, so I'll summarize below and
try to address each idea.

Reviews:
1. Most people can't feel responsible for the whole image, so we need an
option to sign up for responsibility for parts of the image, probably in
groups.
2. There are few topical experts, and some reviews require experts.
3. Who reviews the experts work? especially a collaboration between a
bunch of experts.
4. It sometimes feels silly to review the work of an expert, because of
course its good. 
5. Maybe we should auto accept their stuff.
Testing:
6. Fixes get tested by the person that suffered the bug in the first
place.
7. Informal (untagged) feedback gets lost.
8. Maybe tools could help collect feedback?

First of all, I agree with most of these, and think that whatever we do,
as long as the harvesting process isnt working well, we should try
whatever comes to mind. So while some solutions sound wrong to me, I
definitely think we should do experiments, whether I like them or not. I
w

About 1 - sounds good to me. How do we kick start this? Avi, would you
be willing to take some specific area of Squeak and try to review at
least some portion of patches that come through for it? you mentioned
maybe having groups that review areas. What would you do to start a
group around an area you'd adopt?

2, 3 - note that even in areas where functional dependencies require
experts to tell whether the changes are good or not, a non expert is
able to review for other things. Andreas, for the example of Ned and you
working on morph removal, I think that assuming you'd both looked at all
the code, I wouldn't mind if one of you approved it.

Reading 4 and 5, I feel uncomfortable. Looking at the assumptions behind
the process for minute, here are mine -
A. Everyone is sloppy sometimes, in some way. Inspection by another pair
of eyes helps. So everyones work should be looked at by someone else. 
B. Different people have different standards - this is good, because we
can learn from another, if we bother to look at one anothers code
critically and discuss our findings.
C. What we accept today determines what is posted tommorrow. 

A & B are IMO good reasons to have everyones code looked at at least by
someone. C is something we need to be aware of when considering
optimistic solutions as Marcus is proposing. The worst case scenario of
accepting bad fixes isn't that a bug will get in, it is that we will
drown in a sea of sloppy fixes, caused by a perception that making sure
you got it right is someone elses problem...

So I obviously disagree with 5. I do think we should avoid deadlocking
as much as possible - any two people that fully reviewed a patch fulfill
the review requirement, even if one or both are coauthors. To signal
that they have fully reviewed the code, they should use the [er] tag,
and also comment their review. Anyone think that this is still to
onerous?

On 6/7/8 - I agree. How do we prompt more bug feedback in a way that can
be captured as an [et]? One way that comes to mind is that if people
installed bug fixes off the BFAV, because it was included in their
image, it might seem more natural for them to also give feedback on it
using it.

Daniel

Andreas Raab <andreas.raab at gmx.de> wrote:
> Hi Daniel,
> 
> > Ideas to solve this or alternative explanations would be welcome.
> 
> Let me throw in a few thoughts on this issue. Speaking only for myself I
> find the following issues interesting/relevant in this context:
> 
> * External reviews
> 
> A key problem with external reviews is that you have to know the area
> involved very well to judge the wider implications of changes. It is not
> surprising that typically a few people review certain areas since that's
> their primary area of expertise. One of the key problems in the current
> process is that these people need reviews themselves - even though they are
> experts in the area and an external review is likely to just say "sure it
> works - what did you expect"? Or even worse, there simply isn't anybody who
> feels confident to review that code (who exactly even _could_ review Ned's
> and my joint changes for adding/removing morphs?)
> 
> Here's an interesting observation about myself: I tend to _not_ look at code
> from people I consider "experts" unless I am suspecting something
> non-obvious or outright questionable. In all other cases my basic
> inclination is "go ahead - if there's a problem we can fix that too" rather
> than think ahead and try to foresee what might be affected (which doesn't
> work anyway).
> 
> Personally, there are various people which I would immediately endorse to
> submit fixes and enhancements in various areas. In other words, I trust
> those people enough to be willing to live with their changes without having
> to explicitly review them. And yes, things will break but they'd break with
> or without a review by someone who doesn't really understand it. If so,
> what's the point?
> 
> * External testing
> 
> If you look at the traffic on Squeak-dev, I think you'll find many more
> responses that just say "thanks! that did it" than "[er, et]" or whatever
> tags. The problem is that feedback from people using a fix is mostly
> informal (if at all present). In a perfect world, I would want something
> that I can flag as a fix for some problem. When some person installs the fix
> the person should (later) get asked "did this solve your problem?" with a
> choice of "don't know yet", "yes", "no", "yes but ..." (as in: it fixes the
> problem but shows another one), and "no but ..." (as in: it doesn't but I
> have package Foo and Bar loaded which might interfere).
> 
> One way to solicit this feedback would be to install in particular [FIX]es
> in a way that the system _knows_ this is a fix and can ask the user about it
> after that fix has been in use for a while. Again, here's an example:
> Someone posts a fix for a problem. I load it and keep going. Some time later
> (perhaps when I am about to save/quit the image) the system reminds me that
> I have a bunch of fixes loaded and whether I'd want to comment on them. If I
> choose to, I get the above mentioned options and this feedback is then
> automatically sent to the list.
> 
> The key point about [et] tags for fixes is that people are individually
> bitten by the bugs and that loosing feedback from a single person means
> loosing _all_ of the feedback here. You don't run a [fix] right away unless
> you're bitten by it.
> 
> Cheers,
>   - Andreas
> 
> > -----Original Message-----
> > From: squeakfoundation-bounces at lists.squeakfoundation.org 
> > [mailto:squeakfoundation-bounces at lists.squeakfoundation.org] 
> > On Behalf Of Daniel Vainsencher
> > Sent: Thursday, October 16, 2003 5:57 PM
> > To: squeakfoundation at lists.squeakfoundation.org
> > Cc: ken at kencausey.com; Brent Vukmer; Cees de Groot; Bert Freudenberg
> > Subject: [Squeakfoundation]The Harvesting process and the BFAV
> > 
> > 
> > Hi guys.
> > 
> > The BFAV has become the technical basis for harvesting 
> > progress. In this
> > it has improved the rate of harvesting by removing much of the dirty
> > work that was involved in the old processes. It has improved the
> > efficiency of the approvals many fold.
> > 
> > It has also allowed us to elaborate the process by adding the quality
> > tags, and for the first time having a more or less closed list of
> > patches to be dealt with.
> > 
> > The purpose of the QA tags was to distribute the work of harvesting
> > among more people, by allowing casual participants to help by 
> > reviewing
> > or testing code without accepting the responsibility of the 
> > formal role
> > of harvester. 
> > 
> > As the efficiency of approvals went up, and we have an 
> > in-squeak list of
> > posts and their full status, it has become much more visible that our
> > bottleneck is not approvals. It is very clearly in the 
> > reviewing/testing
> > stage. The reasons are very clear - rather few people help with the
> > reviews/testing, and that quite sporadically. 
> > 
> > This bottleneck is a result of our current structure, and 
> > won't go away
> > by itself. It causes a situation where most patches meant for 
> > the squeak
> > image are ignored. I think this is not likely to last - either we
> > balance this by removing the bottleneck, or contributors of 
> > patches will
> > balance it by posting less, an outcome I find regrettable.
> > 
> > In fact, I think it likely that people are already deciding 
> > not to post
> > fixes they have available because they're likely to be ignored.
> > Therefore, I think that if we can remove the bottleneck, and reach a
> > state where patches submitted are all processed in reasonable time, we
> > will get far more patches, and be able to make much better 
> > progress than
> > we are now. 
> > 
> > I think the following steps are in order -
> > 1. We want to lower the barriers to casual participation. So, 
> > the BFAV,
> > as a prerequisite for participating, should never be absent 
> > for anyone.
> > It should be included in the base image.
> > 2. We should improve and simplify the tools to help 
> > reviewers/testers do
> > their job. 
> > - ClassBoxes, when they're ready, could become a big aid to 
> > testing, by
> > allowing patches to be installed temporarily and then completely
> > removed.
> > - The BFAV should be improved, especially for casual users. Brent, I
> > recommend to open its development so more people can help 
> > with this. The
> > Swiki page is already pretty good, just point to it from the 
> > help in the
> > BFAV. Also it might be good that its future development be carried on
> > using Monticello, to make it easier to create and integrate
> > improvements. 
> > 3. The current flood of BFAV traffic makes people automatically ignore
> > the harvesting process. Therefore it MUST be selectively 
> > moved away from
> > squeak-dev. Cees, or whoever has administrative rights on 
> > lists.sqf.org,
> > can you please create a new mailing list called "squeak-harvest"? This
> > fits Berts existing filters. Having this list would allow some
> > selectivity without losing information.
> > 4. When all is said and done, many people currently won't 
> > participate no
> > matter how low the barriers are made, because of different priorities.
> > Some of this is obviously reasonable, people need to pay the rent and
> > play with whats fun for them, of course. But some of this is IMO not
> > reasonable, and comes from a misperception. I think the NewLook code
> > showcases this very clearly. The voices for its inclusion are 
> > very loud,
> > but nobody bothers to review it, therefore, it isn't included. So I
> > guess that the people doing the requesting somehow fail to 
> > perceive that
> > the condition for inclusion is within their grasp. Of course, I could
> > just be wrong about this. Ideas to solve this or alternative
> > explanations would be welcome.
> > 
> > Daniel
> > _______________________________________________
> > Squeakfoundation mailing list
> > Squeakfoundation at lists.squeakfoundation.org
> > http://lists.squeakfoundation.org/listinfo/squeakfoundation
> > 
> 
> _______________________________________________
> Squeakfoundation mailing list
> Squeakfoundation at lists.squeakfoundation.org
> http://lists.squeakfoundation.org/listinfo/squeakfoundation