From: [...] Daniel Vainsencher Sent: 01 January 2006 19:43
Daniel, what's the date on your machine? Is something off with its conversion to Gregorian somewhere?
It might not be good enough for money handling or legally binding stuff, but it probably is for all kinds of decisions the community is trying to make now.
Quite.
Do you mind updating the minnow page to reflect these requirements?
Done; sorry for the delay. I've hacked with some headings, now that we have more content in there - sorry if that's an issue.
Here I don't quite understand your intention. Do you mean to extend SqP itself?
Ideally.
note that it is not Squeak, so some Squeakers won't hack it directly.
<shrug>. And? A system is a system is a system.
I've CC'ed Cees into this. Cees, what's SqP written in and how easy is it (in your experience) to change?
Anyway, I think the first thing is to make sure we can live with the initial requirements list, and then let move to implementation. For that I was going to ask on the Squeak list for volunteers, unless one of you wants to tackle it yourselves. In the case where someone else is doing it, there's no point in making too many implementation decisions now.
I could do it quickly but not in Squeak, or in Squeak but slowly - I'm not sufficiently familiar with any of the Smalltalk Web frameworks to use them effectively. My day job is almost entirely Microsoft web application hacking (for my sins - but it pays the bills!)
- Peter
On 12/31/05, Peter Crowther Peter@ozzard.org wrote:
<shrug>. And? A system is a system is a system.
I've CC'ed Cees into this. Cees, what's SqP written in and how easy is it (in your experience) to change?
In, err, <cough>, C. The data is in XML, and I have Squeak code to read it so anything that would just use the data for checking 'standing' in SqP could be done in 100% Squeak (I also have "Terminus", a web service, XML/RPC IIRC, that lets you query SqP).
However, now that we're talking about using SqP for elections stuff, my two cents:
KISS.
Claimer: I have lots of experience with organizing and running elections. From real-world municipal elections, to Fidonet factions, Usenet stuff, to open source community stuff. As I said in private mail to you guys, I don't want to meddle around too much with how the Election team operates because formally it should be distinct from the Board and chances are high that i'll find myself being a candidate on a coming election. Having said that...
There are, IMNSHO, two ways to make elections fair: make them extremely transparent, or use technology. It seems that the elections team is mostly discussing the second thing (tell it if I'm wrong, I skimmed the archives but can have gotten a wrong impression from there). What about the first? (and "both", I think, is not a valid answer - election technology often makes everything more complex and therefore almost by definition less transparent).
For example, if an election must be held, setup a website. People see a form, where they can a) fill in their email address, and b) fill in their vote (one or more candidates). And maybe one of these funny images to check they're human :-). In response, the website generates a confirmation mail, with a security hash and the vote. User clicks, vote is registered. At the end of the election, the website automagically stops accepting new votes but displays a list of all email addresses used to vote on the election page instead (and maybe pings squeak-dev to tell everyone the list is up). We have a pretty good grasp of the set of email addresses that form the community. If we suddenly see a whole lot of "strange" mail addresses popping up, something's fishy. If, say, 10% of the voters say the thing is fishy, the election is made invalid. If they don't, the website displays the election results a week later (and probably sends the results to squeak-dev). With the web tools available one could whip up such a system in less than a day.
If every election we hold is canceled because of abuse, that is maybe a time to start thinking about requiring people to register on SqP, or whatever. But until that time, I'd say - keep it as simple as open as possible. Let the community be defined by those who think they're part of the community instead of some artificial social system. I've heard lots of discussions and fears especially during Jini community meetings about the open system, but every vote held since then was ok. No large companies rigging the election by telling all their employees to vote, or something :)
Just my opinion - due to my position I'm not going to defend it or something (at best clarify if I was unclear somewhere - just ask). If SqP will be used, fine. I'm available to help on the work that would be necessary in either case.
Hth,
Cees
It is clear to everyone, I think, that the current set of SqP-certified people does not correspond to the set who should get to vote. So we can't "just use SqP". Many people seem interested in somehow improving the system so that it would generate a reasonable list of members.
Before telling you my reservations, please remember that the object-list approach is about as KISS and as reliable as imaginable. People would sign up on a wiki page if they think they should get a vote, and there would be a time period where people can challenge entries on the page. Assuming everyone plays nice, then within a few weeks -- a month tops -- there would be a list of members that could be frozen as the initial set. This would be a list that everyone should feel good about, because the rules of joining are objective and because the membership applications are all handled publicly.
That said, there are a number of problems with using an automatic reputation system for something as fundamental as membership. People should not go for such a system unless they are comfortable with all of these issues.
A big open question is who are the roots? I asked later but no one has responded. The only objective group I could think of would be to let Alan Kay choose (or, almost equivalently, to make him the only root). Assuming he does not suddenly jump into the fray with the open source community, what else is there? The question has to be answered, and it needs to be answered in a way that does not leave out entire chunks of the community. What do the pro-repsystem people want to choose, here?
Second, have people really thought about what it would be like to live under such a system? Membership would be granted and denied based on a dozen or more off-the-cuff reviews instead of 1-3 careful ones. That's a recipe for superficial decisions -- it will promote populists over quiet doers. Also, reviews would not be archived. People would not have a place to post their *reason* for their certifications. And they cannot post negative reviews at all, right now. (Even if they could, who would put their neck out to do it?) On top of these, don't people worry that a reputation system includes people who pester a lot of members but excludes shier folks who do not want to do that?
Issues aside, there is the question of practical experience. Aside from communities where people pay to join, or where membership is wide open to all comers, practically every community I know of uses the defense-before-committee approach. It is not perfect, but it seems to work well, and it is something that people understand. In particular, it handily addresses all of the issues in the previous paragraph. Is there any experience we can rely on for using reputation systems to define membership? Maybe some of the worries from the previous paragraph do not emerge in practice?
I honestly don't know. I just believe that, aside from evidence to the contrary, we ought to keep it simple and use a tried and true solution.
-Lex
Peter Crowther wrote:
From: [...] Daniel Vainsencher Sent: 01 January 2006 19:43
Daniel, what's the date on your machine? Is something off with its conversion to Gregorian somewhere?
Yes, never mind that, its a problem that will be solved when I give this computer away in the next week or so. For now, don't believe my dates.
About the summary - great, I did some editing to keep the focus on the requirements, and separate it from the rationale for the requirements.
note that it is not Squeak, so some Squeakers won't hack it directly.
<shrug>. And? A system is a system is a system.
Nope - this is the voting system, and it matters that people be able to see it, improve it modify it. Remember we're working on an initial set of requirements, this will probably evolve. So its important that the technology for it is something the community has in common...
I could do it quickly but not in Squeak, or in Squeak but slowly - I'm not sufficiently familiar with any of the Smalltalk Web frameworks to use them effectively. My day job is almost entirely Microsoft web application hacking (for my sins - but it pays the bills!)
I've given a couple of reasons for doing it in Squeak as much as possible, whether you're convinced is something else. I think moving the trust flow algorithm to Squeak would be a good way for us to learn more about it and play with variants. About the voting system itself, maybe you can find a Seaside veteran to pair with in the list...
Just ideas :-)
Daniel
First, let me remind people that the general proposal I'm pushing is used extraordinarily widely. The object-list, review-committee approach is likely as old as human memory. It's at least as old as martial arts schools. Further, it's just about the only approach used where there is a non-trivial review at all. All other communities I can think of either base membership on money, or allow anyone to be a member. If you think you see a trivial dismissal, then for the sake of honest argument you should consider again more carefully.
Let's get into some detail, now. Daniel has the following main sticking points: how verification would work, the importance of public archived reviews, and the importance of negative comments. Let me comment on those, and then summarize where things seem to stand on reputation systems in general.
It is certainly true that the objective-list process requires verification. Any process does. I don't see why this is a problem. I, at least, will challenge people who post junk. Further, anyone who does not have any positive verifications is certainly not going to get a vote.
I would suggest that we have some sort of registration committee that is responsible for doing the final judgement calls. We could even use Squeak People ranks for that, since it is not important to get the registration committee exactly representative. They can decide among themselves how much to trust each possible reviewer's verifications.
Now, yes, it is possible for the registration folks to be dishonest. However, they must do everything publicly. If they are dishonest, then the vote is not widely respected and will fail. This situation is common to most any election system. At some point somebody has to count the votes, and if they insist on publicly doing it incorrectly, then the vote doesn't happen. They cannot, however rig the vote in their favor, if everything is done publicly and with carefully drawn rules. So long as the object list of requirements is indeed objective, there should be no wiggle room for registration tsars to affect the vote.
That's the verification issue. The other questions of archiving comments and of negative comments go hand in hand. Negative comments are important in the rare case that someone does try to hack themselves a vote. And archiving negative comments is critical in cases where there is something tricky about the application. For example, suppose someone links to some code that turns out to have mostly been copied from someone else. The dual measures of negative comments and archived application processes means that these situations can be solved, thus improving our general confidence that the official membership designations are accurate.
Now let's review reputation systems in general. There are important issues that need to be answered if we go with one to choose membership.
First, I think we all agree now that we can't just use the current Squeak People rankings, correct? As recently as Jan. 1, Daniel proposed that we do just that. Hopefully that sub-proposal is now dead?
That sub-proposal, however, was the simple option. If we don't follow that, then the question opens of how to fix things up. The most important question here is the choice of roots. To put it politely, it is a bad idea for a subgroup to declare themselves leaders and then also name themselves the as the source of membership for future elections. Can we please agree on this in the abstract and not dig into individual cases?
Leaving aside that option, who should we pick as roots? Answering that question seems just as hard as holding an election, bringing us back to the beginning.
Second, reviews matter, but our current system doesn't give us very good ones. Squeak People reviews are generally "oh, I know that guy, I think he's a so and so". Hopefully there would at least be a lot of these, but Daniel has pointed out that in many cases there are only 2-3 that determine a person's membership. The objective-list approach would mean that there is an objective set of criteria that are used in deciding who is a member. At least 1 person would carefully review each new applicant against that list. Further, the list would be public and could be talked about, instead of every person promoting whoever they feel like.
Third, why do we have reason to expect the current reputation system (or any rep system) to generate accurate membership desginations? It is now widely agreed that many of the people who should get votes are not currently given votes by Squeak People. Why, exactly, should we think that it will work in the future? The objective-list approach is old and provides a lot of positive experience for us. What can give us similar confidence in the reputation-system approach?
Finally, if indeed the system can be modified to answer he above questions (e.g., requiring reviews to be attached to certs), then is the result truly simple any longer? The main appeal of a reputation system for most people appears to be that it is simple. However, it's only simple if we do it poorly. Doing it well appears just at least as complicated, and seems to require just as much raw effort, as a traditional objective-list approach.
Overall, I like reputation systems and hope to see more experiments with them. This is not the time for an experiment, though. Bylaws should be boring.
Lex Spoon
Before we start the point by point phase, something positive:
Lex Spoon wrote:
Issues aside, there is the question of practical experience. Aside from communities where people pay to join, or where membership is wide open to all comers, practically every community I know of uses the defense-before-committee approach. It is not perfect, but it seems to work well, and it is something that people understand. In particular, it handily addresses all of the issues in the previous paragraph. Is there any experience we can rely on for using reputation systems to define membership? Maybe some of the worries from the previous paragraph do not emerge in practice?
I honestly don't know. I just believe that, aside from evidence to the contrary, we ought to keep it simple and use a tried and true solution.
I don't know, I think SqP can save us lots of ugly work by making trust transitive. Its already implemented. It seems to have built in safeguard limiting the amount of damage a single person can do. And I sort of like the fact that its new, heck, we might learn something from it.
It is clear to everyone, I think, that the current set of SqP-certified people does not correspond to the set who should get to vote.
Agreed.
So we can't "just use SqP".
This does not follow - creating new accounts on SqP is almost as easy as adding your name to a wiki page. Do you already have a wiki page with all the voter names?
Converting SqP accounts to voters is much easier than converting names-on-a-wiki into voters - SqP already has the interfaces for certification, query for voter list and so forth. A wiki starts with no structure. If you add it up front, that's work to be done. If you don't, you'll get a mess, and work harder later.
Many people seem interested in somehow improving the system so that it would generate a reasonable list of members.
There is currently no evidence that any code improvements are necessary at all. It is quite possible that the only thing it lacks is use, and when people know that it matters, they'll certify.
Lex, all of the above, I've said about 3 times on this list now. Are you not seeing this?
Before telling you my reservations, please remember that the object-list approach is about as KISS and as reliable as imaginable.
Lets see...
People would sign up on a wiki page if they think they should get a vote,
Simple so far.
and there would be a time period where people can challenge entries on the page.
So by default, everyone gets in, because if I don't know someone, I'm certainly not going to call him a lier, right?
Assuming everyone plays nice, then within a few weeks -- a month tops -- there would be a list of members that could be frozen as the initial set.
Whoa! when did that happen? what usually happens on a wiki page is that any initial structure is used, overloaded and then starts crumbling. In this case, I expect a list of people that put themselves up and are ignored because nobody knows them. Some of them put up excuses that someone chooses to challenge by asking a couple of questions. Half of those questions are answered. Which of those entries are in? who decides that?
This would be a list that everyone should feel good about, because the rules of joining are objective and because the membership applications are all handled publicly.
So far we have a wiki page with persons names, and a list of rules about their accomplishments. How are these matched? do people list their accomplishments? do they provide a link so people can verify it? do they just write the number of the rule that they believe they should be accepted under? I'm not sure what you mean by someone "feeling good" about it. I get the feeling that membership is so open as to be meaningless. Anyone can write themselves in, verification is usually impossible, therefore won't be done. The applications happen publically, but I'm not at all sure in what sense they are handled - seems like they are simply collected.
That said, there are a number of problems with using an automatic reputation system for something as fundamental as membership. People should not go for such a system unless they are comfortable with all of these issues.
A big open question is who are the roots? I asked later but no one has responded. The only objective group I could think of would be to let Alan Kay choose (or, almost equivalently, to make him the only root).
In what sense you call that objective?
Assuming he does not suddenly jump into the fray with the open source community, what else is there?
First - what already exists. True, this is an arbitrary set of roots, but until we find out that someone that has an account, and is a serious user of Squeak can't get certified because the roots are insufficient, there is no problem.
That said...
The question has to be answered, and it needs to be answered in a way that does not leave out entire chunks of the community. What do the pro-repsystem people want to choose, here?
I think that the criterion for adding roots should be "heads of communities" (AK is the prime example, of course). I think this criterion should be judged by votes of the existing voter body.
Second, have people really thought about what it would be like to live under such a system? Membership would be granted and denied based on a dozen or more off-the-cuff reviews instead of 1-3 careful ones.
I don't know where you are taking these assumptions. Numerically, very few certifications seem to be sufficient: only KenC and I have certified Peter, and he got the certification we gave him. As to the quality of the reviews, why exactly should I review more carefully on a wiki page than on SqP?
That's a recipe for superficial decisions -- it will promote populists over quiet doers. Also, reviews would not be archived. People would not have a place to post their *reason* for their certifications.
Of course they do - SqP provides you with some web space, and from a person I can go to the pages of his certifiers. The "reviews" would be archived whenever someone bothers to write them (probably rarely). Note that explicit reviews are really not that important an idea in a web of trust. The transitivity takes care of it. If you trust me enough to read my reviews, just certify me, and the juice will pass on to those I certify.
And they cannot post negative reviews at all, right now.
They can. No negative certifications, that's all. But why do we need those?
(Even if they could, who would put their neck out to do it?)
If I know some new comer is a cheat, why shouldn't I mail his certifiers and mention the problem? assuming I have trust from people that are not dependent on his certifiers, what's to stop me?
On top of these, don't people worry that a reputation system includes people who pester a lot of members but excludes shier folks who do not want to do that?
If I get pestered, I might certify someone who I'd otherwise ignore, but you think I would certify them high? and can you please explain in comparison what would happen in a wiki based system? sounds to me I can pester people to vouch for me in any medium.
Daniel
Lex Spoon wrote: [snip]
First, I think we all agree now that we can't just use the current Squeak People rankings, correct? As recently as Jan. 1, Daniel proposed that we do just that. Hopefully that sub-proposal is now dead?
Oh, that never existed except as a miscommunication. I meant "current" to refer to the time of the vote, not to the time of the mail.
That sub-proposal, however, was the simple option. If we don't follow that, then the question opens of how to fix things up. The most important question here is the choice of roots.
[snip]
Can we please agree on this in the abstract and not dig into individual cases?
Sure, I propose we start with the current (somewhat arbitrary) roots, and extend them only by vote, based on some abstract criteria.
Leaving aside that option, who should we pick as roots? Answering that question seems just as hard as holding an election, bringing us back to the beginning.
Nah, I think that choosing roots is much more like choosing vote verifiers in your proposal - something that is not that important, because for any reasonable choice you will get reasonable results, and the current status is not that unreasonable. Except that roots don't have to work nearly as hard as reviewers... :-)
[snip some objections to reputation systems and SqP specifically]
Overall, I like reputation systems and hope to see more experiments with them. This is not the time for an experiment, though. Bylaws should be boring.
Well, we're not yet proposing to put any voting system into any bylaws. I think Ken is right in proposing we try this system for some upcoming decision (maybe, but not necessarily, the choice of SqF board), and rethink our opinions when experience has made us wiser. What do you think?
If we decide SqP is unworkable, we can always start choosing voters by Mantis later on.
Daniel
elections@lists.squeakfoundation.org
-
Cees De Groot -
Daniel Vainsencher -
Lex Spoon -
Peter Crowther