One of my goals for 3.10 is to improve the quality of the image. Our first release (coming soon!) will have only green tests, and each following release will have only green tests. But there are many other things that could be checked automatically. For example, there should be no unimplemented methods in the released image. Unfortunately, there are a lot right now, so we can't make that rule. But I would like to have all these fixed by the end of the 3.10 cycle and to be able to enforce the rule that no release has any unimplemented methods.
Jerome Peace has been working on getting rid of unimplemented methods and has a lot of fixes. You can find them at http://bugs.impara.de/view.php?id=4544 This is the original Mantis issue that he has been working on. Most of the fixes are in "child" issues, but you can find them from that page.
You can help by checking Jerome's fixes. If you are familiar with the code he is changing, read it and see whether you can spot anything wrong. If you can, post a note. If you can't, please post a note that, as far as you can tell, it looks good. If you aren't that familiar with the code, but are working on an application that uses it, please file in the changes and try out your application. Again, report on the results!
If two or three people try out some changes and everybody thinks they are OK, the release team will mark the issue as "resolved" and put it in the next release.
We will make sure the code doesn't break any tests. But if you don't try out the code then we'll have to try it, as well, and that will take a lot more time. So, you can help us get more work done by checking out these fixes.
Thanks!
-Ralph Johnson
El 1/26/07 10:10 AM, "Ralph Johnson" johnson@cs.uiuc.edu escribió:
Jerome Peace has been working on getting rid of unimplemented methods and has a lot of fixes. You can find them at http://bugs.impara.de/view.php?id=4544 This is the original Mantis issue that he has been working on. Most of the fixes are in "child" issues, but you can find them from that page.
You can help by checking Jerome's fixes. If you are familiar with the code he is changing, read it and see whether you can spot anything wrong. If you can, post a note. If you can't, please post a note that, as far as you can tell, it looks good. If you aren't that familiar with the code, but are working on an application that uses it, please file in the changes and try out your application. Again, report on the results!
I have to look again, but what I when discuss the subject with Jerome, works.
Pavel have few unimplemented in his Kernel, but I don't have his code.
Maybe all Pavel and Jerome work about unimplemented could give a cleaner image.
Edgar
__________________________________________________ Preguntá. Respondé. Descubrí. Todo lo que querías saber, y lo que ni imaginabas, está en Yahoo! Respuestas (Beta). ¡Probalo ya! http://www.yahoo.com.ar/respuestas
2007/1/26, Ralph Johnson johnson@cs.uiuc.edu:
One of my goals for 3.10 is to improve the quality of the image. Our first release (coming soon!) will have only green tests, and each following release will have only green tests.
How does removing failing tests improve the quality?
Cheers Philippe
But there are many other things that could be checked automatically. For example, there should be no unimplemented methods in the released image. Unfortunately, there are a lot right now, so we can't make that rule. But I would like to have all these fixed by the end of the 3.10 cycle and to be able to enforce the rule that no release has any unimplemented methods.
Jerome Peace has been working on getting rid of unimplemented methods and has a lot of fixes. You can find them at http://bugs.impara.de/view.php?id=4544 This is the original Mantis issue that he has been working on. Most of the fixes are in "child" issues, but you can find them from that page.
You can help by checking Jerome's fixes. If you are familiar with the code he is changing, read it and see whether you can spot anything wrong. If you can, post a note. If you can't, please post a note that, as far as you can tell, it looks good. If you aren't that familiar with the code, but are working on an application that uses it, please file in the changes and try out your application. Again, report on the results!
If two or three people try out some changes and everybody thinks they are OK, the release team will mark the issue as "resolved" and put it in the next release.
We will make sure the code doesn't break any tests. But if you don't try out the code then we'll have to try it, as well, and that will take a lot more time. So, you can help us get more work done by checking out these fixes.
Thanks!
-Ralph Johnson
On Jan 26, 2007, at 16:03 , Philippe Marschall wrote:
2007/1/26, Ralph Johnson johnson@cs.uiuc.edu:
One of my goals for 3.10 is to improve the quality of the image. Our first release (coming soon!) will have only green tests, and each following release will have only green tests.
How does removing failing tests improve the quality?
Woa, where does that hostility come from? There is another way to ensure all tests are green, besides removing the failing ones.
- Bert -
But there are many other things that could be checked automatically. For example, there should be no unimplemented methods in the released image. Unfortunately, there are a lot right now, so we can't make that rule. But I would like to have all these fixed by the end of the 3.10 cycle and to be able to enforce the rule that no release has any unimplemented methods.
Jerome Peace has been working on getting rid of unimplemented methods and has a lot of fixes. You can find them at http://bugs.impara.de/view.php?id=4544 This is the original Mantis issue that he has been working on. Most of the fixes are in "child" issues, but you can find them from that page.
You can help by checking Jerome's fixes. If you are familiar with the code he is changing, read it and see whether you can spot anything wrong. If you can, post a note. If you can't, please post a note that, as far as you can tell, it looks good. If you aren't that familiar with the code, but are working on an application that uses it, please file in the changes and try out your application. Again, report on the results!
If two or three people try out some changes and everybody thinks they are OK, the release team will mark the issue as "resolved" and put it in the next release.
We will make sure the code doesn't break any tests. But if you don't try out the code then we'll have to try it, as well, and that will take a lot more time. So, you can help us get more work done by checking out these fixes.
Thanks!
-Ralph Johnson
Some of the bugs got fixed, some of the tests were wrong and got deleted, and the rest of the tests got moved to a to-do list.
It is important that people be able to run the tests to see if they broke anything. This cannot happen if tests that are known to fail are mixed with tests that are supposed to succeed.
Perhaps you have never worked on a project in which all tests worked all the time. This is an important tool in achieving high quality software.
-Ralph Johnson
2007/1/26, Bert Freudenberg bert@freudenbergs.de:
On Jan 26, 2007, at 16:03 , Philippe Marschall wrote:
2007/1/26, Ralph Johnson johnson@cs.uiuc.edu:
One of my goals for 3.10 is to improve the quality of the image. Our first release (coming soon!) will have only green tests, and each following release will have only green tests.
How does removing failing tests improve the quality?
Woa, where does that hostility come from? There is another way to ensure all tests are green, besides removing the failing ones.
What hostility? I could not see why this improves the quality because to me the first step to fix a problem is to admit that you have a problem. Failing tests are pointer to problems for me. Removing failing tests because they can not be fixed today or tomorrow looked to me like an attempt to hide hide a problem. So I asked and now I know the reason why it was done.
Philippe
But there are many other things that could be checked automatically. For example, there should be no unimplemented methods in the released image. Unfortunately, there are a lot right now, so we can't make that rule. But I would like to have all these fixed by the end of the 3.10 cycle and to be able to enforce the rule that no release has any unimplemented methods.
Jerome Peace has been working on getting rid of unimplemented methods and has a lot of fixes. You can find them at http://bugs.impara.de/view.php?id=4544 This is the original Mantis issue that he has been working on. Most of the fixes are in "child" issues, but you can find them from that page.
You can help by checking Jerome's fixes. If you are familiar with the code he is changing, read it and see whether you can spot anything wrong. If you can, post a note. If you can't, please post a note that, as far as you can tell, it looks good. If you aren't that familiar with the code, but are working on an application that uses it, please file in the changes and try out your application. Again, report on the results!
If two or three people try out some changes and everybody thinks they are OK, the release team will mark the issue as "resolved" and put it in the next release.
We will make sure the code doesn't break any tests. But if you don't try out the code then we'll have to try it, as well, and that will take a lot more time. So, you can help us get more work done by checking out these fixes.
Thanks!
-Ralph Johnson
Philippe Marschall wrote:
2007/1/26, Bert Freudenberg bert@freudenbergs.de:
On Jan 26, 2007, at 16:03 , Philippe Marschall wrote:
2007/1/26, Ralph Johnson johnson@cs.uiuc.edu:
One of my goals for 3.10 is to improve the quality of the image. Our first release (coming soon!) will have only green tests, and each following release will have only green tests.
How does removing failing tests improve the quality?
Woa, where does that hostility come from? There is another way to ensure all tests are green, besides removing the failing ones.
What hostility? I could not see why this improves the quality because to me the first step to fix a problem is to admit that you have a problem. Failing tests are pointer to problems for me. Removing failing tests because they can not be fixed today or tomorrow looked to me like an attempt to hide hide a problem. So I asked and now I know the reason why it was done.
Philippe
Philippe, where did you read that failing tests will be removed? "First release will have only green tests" means, that all tests remain and will pass, not fail. There will be no test removal at all! I'm, pretty sure you misunderstood something.
Elod
But there are many other things that could be checked automatically. For example, there should be no unimplemented methods in the released image. Unfortunately, there are a lot right now, so we can't make that rule. But I would like to have all these fixed by the end of the 3.10 cycle and to be able to enforce the rule that no release has any unimplemented methods.
Jerome Peace has been working on getting rid of unimplemented methods and has a lot of fixes. You can find them at http://bugs.impara.de/view.php?id=4544 This is the original Mantis issue that he has been working on. Most of the fixes are in "child" issues, but you can find them from that page.
You can help by checking Jerome's fixes. If you are familiar with the code he is changing, read it and see whether you can spot anything wrong. If you can, post a note. If you can't, please post a note that, as far as you can tell, it looks good. If you aren't that familiar with the code, but are working on an application that uses it, please file in the changes and try out your application. Again, report on the results!
If two or three people try out some changes and everybody thinks they are OK, the release team will mark the issue as "resolved" and put it in the next release.
We will make sure the code doesn't break any tests. But if you don't try out the code then we'll have to try it, as well, and that will take a lot more time. So, you can help us get more work done by checking out these fixes.
Thanks!
-Ralph Johnson
Hi folks!
Philippe Marschall wrote:
2007/1/26, Bert Freudenberg bert@freudenbergs.de:
On Jan 26, 2007, at 16:03 , Philippe Marschall wrote:
2007/1/26, Ralph Johnson johnson@cs.uiuc.edu:
One of my goals for 3.10 is to improve the quality of the image.
Our
first release (coming soon!) will have only green tests, and each following release will have only green tests.
How does removing failing tests improve the quality?
Woa, where does that hostility come from? There is another way to ensure all tests are green, besides removing the failing ones.
etc etc.
Ok, IMHO this is all about classification of tests. Perhaps someone has already proposed the following but how about:
- If there is a bug and someone authors a test to show it and it enters the image (lets ignore *that* particular question for 3 seconds - many of course argue that it should not enter the image unless accompanied with a fix), why not mark it as "has never worked" - or something along those lines?
That way we can keep all *working* (not marked) tests green and at the same time have a list of non working tests (those marked) that typically are all red. When something is fixed and turned green we remove the marker.
Sure, this may be a daft idea :), just wanted to mention it.
regards, Göran
Dear Goran,
I am with you on this one. Having broken tests visible, but correctly categorised is key to getting things sorted IMHO. I dont think that hiding things away on mantis is not visible enough.
You might like to try my improvements to TestRunner and SUnit which implement these changes. You can try it by executing.
Installer fixBug: 5639.
best regards
Keith
Ok, IMHO this is all about classification of tests. Perhaps someone has already proposed the following but how about:
- If there is a bug and someone authors a test to show it and it enters
the image (lets ignore *that* particular question for 3 seconds - many of course argue that it should not enter the image unless accompanied with a fix), why not mark it as "has never worked" - or something along those lines?
That way we can keep all *working* (not marked) tests green and at the same time have a list of non working tests (those marked) that typically are all red. When something is fixed and turned green we remove the marker.
Sure, this may be a daft idea :), just wanted to mention it.
regards, Göran
____________________________________________________
Yahoo! Photos is now offering a quality print service from just 7p a photo. http://uk.photos.yahoo.com
I actually side with Ralph on this one. It is very satisfying to see the test runner turn green. With tests in the image that you cannot fix, you will never get this satisfaction.
You should get used to using the bug tracker. If you have a little time to spare, go there, pick an issue that looks interesting or easy, fix it. Or better yet, write a test for it if there is none, run it, then fix it, and enjoy the soothing green of the test run. It'll make you smile :)
- Bert -
On Jan 29, 2007, at 18:11 , Keith Hodges wrote:
Dear Goran,
I am with you on this one. Having broken tests visible, but correctly categorised is key to getting things sorted IMHO. I dont think that hiding things away on mantis is not visible enough.
You might like to try my improvements to TestRunner and SUnit which implement these changes. You can try it by executing.
Installer fixBug: 5639.
best regards
Keith
Ok, IMHO this is all about classification of tests. Perhaps someone has already proposed the following but how about:
- If there is a bug and someone authors a test to show it and it
enters the image (lets ignore *that* particular question for 3 seconds - many of course argue that it should not enter the image unless accompanied with a fix), why not mark it as "has never worked" - or something along those lines?
That way we can keep all *working* (not marked) tests green and at the same time have a list of non working tests (those marked) that typically are all red. When something is fixed and turned green we remove the marker.
Sure, this may be a daft idea :), just wanted to mention it.
regards, Göran
Bert Freudenberg wrote:
I actually side with Ralph on this one. It is very satisfying to see the test runner turn green. With tests in the image that you cannot fix, you will never get this satisfaction.
Yes you can, you select "all standard tests" and you select the filter, "known issues", run the tests and you get all green for the tests that are supposed to be green. There are also filters for tests that are not expected to work on this platform/vm release etc
I personally would like to fill the (or should I say an) image with 100's of test stubs saying , we need a test for this and for that.
Keith
___________________________________________________________ All new Yahoo! Mail "The new Interface is stunning in its simplicity and ease of use." - PC Magazine http://uk.docs.yahoo.com/nowyoucan.html
I personally would like to fill the (or should I say an) image with 100's of test stubs saying , we need a test for this and for that.
it would be a nice way to invite people participate to the effort. This is now years that we are pushing test writing... and we should continue.
Stef
I agree too. This was not our intention to ship 3.9 with broken tests but we had to stop and nobody really stepped up to help (except one person but I do not remember and we tried to harvest his fixes).
Now in VW a guy in our team extended SUnitToo to support failing tests and I think that this is a nice alternative.
PS: I would avoid to put code on mantis as cs but publish packages on squeaksource since this is easier to deal with them. Create one package FailingTest and this way if someone wants to have a look to fix a test he can load all of them in one shot.
On 29 janv. 07, at 18:23, Bert Freudenberg wrote:
I actually side with Ralph on this one. It is very satisfying to see the test runner turn green. With tests in the image that you cannot fix, you will never get this satisfaction.
stephane ducasse a écrit :
I agree too. This was not our intention to ship 3.9 with broken tests but we had to stop and nobody really stepped up to help (except one person but I do not remember and we tried to harvest his fixes).
Now in VW a guy in our team extended SUnitToo to support failing tests and I think that this is a nice alternative.
PS: I would avoid to put code on mantis as cs but publish packages on squeaksource since this is easier to deal with them. Create one package FailingTest and this way if someone wants to have a look to fix a test he can load all of them in one shot.
On 29 janv. 07, at 18:23, Bert Freudenberg wrote:
I actually side with Ralph on this one. It is very satisfying to see the test runner turn green. With tests in the image that you cannot fix, you will never get this satisfaction.
If I follow dominating logic in this thread, the process to incorporate a patch could be: 1) make a subclass of KnownBug instead of TestCase for the bug test. 2) write or Load a patch. 3) check non regression of all TestCase suite (green bar). 4) reject the patch or goto 2) if TestCase suite bar is red 5) check correction of targeted KnownBug 6) reject the patch or goto 2) if bug test red 7) accept the patch and move test from KnownBug to TestCase hierarchy
As you can see, you will need an image holding both KnwonBugs and TestCase during patch process. In this image, KnownBugs must not turn TestCase suite bar red. So the SUnit framework SHALL deal with this feature.
Alternative is to use two images, one for for bugfix and the other for non regression test, but with the risk of having images diverging.
Whatever you use, protocol or subclass or any other flag somewhere in the image, you just need a classification that you can change easily.
If the SUnit framework deal with this, the question raised is why should we remove KnownBugs from the image? - for cosmetic reasons? (it cannot be a commercial reason: as already said, knowing some big company success, it's obviously not so hard to sell bugs) - for cleaning reasons? having a small image... I cannot believe that KnwoBugs suite is as big as TestCase suite, so you gain relatively few.
Alternatively, why should we keep KnownBugs in the image? - for encouraging people to fix them? people look more often in their image than in mantis don't they? Scanning the mailing list, you will see that a lot of people tried to make these light turn green in 3.9 process, not always successfully that's true, some bugs are terse.
If we keep it in the image, it must be a package maintained with SqueakMap and Monticello so that people can share. I totally agree with Steph.
If we do not, we have Mantis database. Mantis is rich but information is rather scattered... It contains old bugs already fixed, complete and incomplete fixes, enhancements, feature requests and non bug.
It is not convenient enough and we need an improved way to load all known bugs at once. I am sure Steph and Marcus can confirm that.
However, if one wants to fix a bug, he should better keep the link on mantis database as it is the place to read other programmers discussion, bug tests and eventual (partial) fix attempts...
So this is the only thing that annoy me in Steph proposition. How to re-concile these two forms?
It must be noted that a side effect of 3.9 red light has been that bugs were reported several times in mantis with several patches uncompatible or uncomplete...
Use an automaton with some special tags inserted in mantis comments processed within Squeak? Human usable Hyperlink to mantis from within squeak?
Nicolas
On Jan 29, 2007, at 9:23 AM, Bert Freudenberg wrote:
I actually side with Ralph on this one. It is very satisfying to see the test runner turn green. With tests in the image that you cannot fix, you will never get this satisfaction.
True, the green bar is very satisfying. But Ralph's reasons go a bit deeper. The purpose of a test suite is to provide immediate feedback on during development. Running the suite is like asking "how am I doing?" A green bar means "fine," and a red bar means, "STOP! You just introduced a bug!" The value of a test suite is that it can let us know that we've introduced a bug at the moment it happens. That's the best time to fix it, because the person who understands it best is right there and has all the information he needs.
On the other hand, a test suite is *not* a bug database. That's what Mantis is for. "Visibility" of known bugs doesn't get them fixed, it just makes them overshadow the unknown bugs we'll introduce going forward.
Colin
On 2007 January 29 16:58, Colin Putney wrote:
On Jan 29, 2007, at 9:23 AM, Bert Freudenberg wrote:
I actually side with Ralph on this one. It is very satisfying to see the test runner turn green. With tests in the image that you cannot fix, you will never get this satisfaction.
True, the green bar is very satisfying. But Ralph's reasons go a bit deeper. The purpose of a test suite is to provide immediate feedback on during development. Running the suite is like asking "how am I doing?" A green bar means "fine," and a red bar means, "STOP! You just introduced a bug!" The value of a test suite is that it can let us know that we've introduced a bug at the moment it happens.
Yes, also I think this is important when test are automated (e.g using a test server) - add a change to the image (install a new version of MCZ, load a changeset) and run tests: If all succeed, good, if something fails, a bug was introduced. Unless something in the system keeps track of bugs that are "expected to fail" and compares them to what just happened, it is important to have all tests succeed for auto-testing to work.
Milan
That's the best time to fix it, because the person who understands it best is right there and has all the information he needs.
On the other hand, a test suite is *not* a bug database. That's what Mantis is for. "Visibility" of known bugs doesn't get them fixed, it just makes them overshadow the unknown bugs we'll introduce going forward.
Colin
Milan Zimmermann wrote:
On 2007 January 29 16:58, Colin Putney wrote:
On Jan 29, 2007, at 9:23 AM, Bert Freudenberg wrote:
I actually side with Ralph on this one. It is very satisfying to see the test runner turn green. With tests in the image that you cannot fix, you will never get this satisfaction.
True, the green bar is very satisfying. But Ralph's reasons go a bit deeper. The purpose of a test suite is to provide immediate feedback on during development. Running the suite is like asking "how am I doing?" A green bar means "fine," and a red bar means, "STOP! You
Yes except that the current suite of tests take ages to run, and so does not fulfil this goal.
The solution which already exists! Is to categorise the tests, so that a suite of short tests can be run more frequently in less than 2 minutes.
If you have this categorisation then it is trivial to have a category of 'known issues', which you simply do not run if you want to see your green light.
The improved TestRunner, times each test, and can automatically sort the long from the short, the network using tests from the non-network using tests etc.
Not forgetting that there are other categories of tests needed in order to get that all hallowed green bar, such as: tests that I would not expect to work on my platform, or tests that will not work on the vm that I am using, and tests that will not work in this version of the image.
Pulling your known issues tests into a separate class, is not a good solution because then you loose the ability to subclass from TestCase sensibly.
Pulling your known issues into a separate package is not a good solution, because then you loose the context for those tests, since they belong in the same context as those tests which pass. You need that context if you are ever going to fix them.
Finally, breaking things up physically, rather than tagging things 'mentally' so to speak ruins any kind of smooth workflow. Write test fix test, becomes write test in one place, when it works move it to another place, debug it again in the new context.
To try the improved TestRunner, try
Installer fixBug: 5639.
best regards
Keith
___________________________________________________________ All new Yahoo! Mail "The new Interface is stunning in its simplicity and ease of use." - PC Magazine http://uk.docs.yahoo.com/nowyoucan.html
Pulling your known issues tests into a separate class, is not a good solution because then you loose the ability to subclass from TestCase sensibly.
Pulling your known issues into a separate package is not a good solution, because then you loose the context for those tests, since they belong in the same context as those tests which pass. You need that context if you are ever going to fix them.
Finally, breaking things up physically, rather than tagging things 'mentally' so to speak ruins any kind of smooth workflow. Write test fix test, becomes write test in one place, when it works move it to another place, debug it again in the new context.
I have done this for a long time. Putting nonworking tests into separate classes and packages is in fact perfectly fine. My experience is that your arguments are wrong.
Moving code around in Smalltalk is very easy. It is almost as easy to move a method from one class to another as it is to press a button. It takes MUCH longer to figure out whether you want to move it. Tests should be independent of each other, so moving it should make no difference to debugging.
The big advantage of SUnit is that it is extremely simple. Simple things work. Sometimes complicated things work, too, but it is harder!
-Ralph Johnson
Keith Hodges wrote:
Dear Goran,
I am with you on this one. Having broken tests visible, but correctly categorised is key to getting things sorted IMHO. I dont think that hiding things away on mantis is not visible enough.
Just to clarify, we will have the ability to release more than one image, a 'refined', all tests green, and slimmed down image for public consumption, and a 'this is what needs working on' image for those interested in fixing things.
Keith
___________________________________________________________ Try the all-new Yahoo! Mail. "The New Version is radically easier to use" The Wall Street Journal http://uk.docs.yahoo.com/nowyoucan.html
2007/1/29, Elod Kironsky kironsky@grisoft.cz:
Philippe Marschall wrote:
2007/1/26, Bert Freudenberg bert@freudenbergs.de:
On Jan 26, 2007, at 16:03 , Philippe Marschall wrote:
2007/1/26, Ralph Johnson johnson@cs.uiuc.edu:
One of my goals for 3.10 is to improve the quality of the image. Our first release (coming soon!) will have only green tests, and each following release will have only green tests.
How does removing failing tests improve the quality?
Woa, where does that hostility come from? There is another way to ensure all tests are green, besides removing the failing ones.
What hostility? I could not see why this improves the quality because to me the first step to fix a problem is to admit that you have a problem. Failing tests are pointer to problems for me. Removing failing tests because they can not be fixed today or tomorrow looked to me like an attempt to hide hide a problem. So I asked and now I know the reason why it was done.
Philippe
Philippe, where did you read that failing tests will be removed? "First release will have only green tests" means, that all tests remain and will pass, not fail. There will be no test removal at all! I'm, pretty sure you misunderstood something.
http://bugs.impara.de/view.php?id=5527
Philippe
Elod
But there are many other things that could be checked automatically. For example, there should be no unimplemented methods in the released image. Unfortunately, there are a lot right now, so we can't make that rule. But I would like to have all these fixed by the end of the 3.10 cycle and to be able to enforce the rule that no release has any unimplemented methods.
Jerome Peace has been working on getting rid of unimplemented methods and has a lot of fixes. You can find them at http://bugs.impara.de/view.php?id=4544 This is the original Mantis issue that he has been working on. Most of the fixes are in "child" issues, but you can find them from that page.
You can help by checking Jerome's fixes. If you are familiar with the code he is changing, read it and see whether you can spot anything wrong. If you can, post a note. If you can't, please post a note that, as far as you can tell, it looks good. If you aren't that familiar with the code, but are working on an application that uses it, please file in the changes and try out your application. Again, report on the results!
If two or three people try out some changes and everybody thinks they are OK, the release team will mark the issue as "resolved" and put it in the next release.
We will make sure the code doesn't break any tests. But if you don't try out the code then we'll have to try it, as well, and that will take a lot more time. So, you can help us get more work done by checking out these fixes.
Thanks!
-Ralph Johnson
Philippe Marschall wrote:
2007/1/29, Elod Kironsky kironsky@grisoft.cz:
Philippe Marschall wrote:
2007/1/26, Bert Freudenberg bert@freudenbergs.de:
On Jan 26, 2007, at 16:03 , Philippe Marschall wrote:
2007/1/26, Ralph Johnson johnson@cs.uiuc.edu:
One of my goals for 3.10 is to improve the quality of the
image. Our
first release (coming soon!) will have only green tests, and each following release will have only green tests.
How does removing failing tests improve the quality?
Woa, where does that hostility come from? There is another way to ensure all tests are green, besides removing the failing ones.
What hostility? I could not see why this improves the quality because to me the first step to fix a problem is to admit that you have a problem. Failing tests are pointer to problems for me. Removing failing tests because they can not be fixed today or tomorrow looked to me like an attempt to hide hide a problem. So I asked and now I know the reason why it was done.
Philippe
Philippe, where did you read that failing tests will be removed? "First release will have only green tests" means, that all tests remain and will pass, not fail. There will be no test removal at all! I'm, pretty sure you misunderstood something.
http://bugs.impara.de/view.php?id=5527
Philippe
Sorry Philippe, then I have to agree with you and join to Goran's proposition to classify the test, removing them is not a good solution I think.
Elod
Failing tests do not belong in the image. They belong in Mantis. If you have a test that ought to run but does not then post it to Mantis.
We should not release an image with failing tests. In general, the 3.10 release team will not accept changes that break tests. If the image contains failing tests then it is hard for people to tell that their proposed changed breaks them. It is crucial that there be NO failing tests in the image so that anyone can easily check whether their change caused any of the tests to fail. Go to the test runner, select all tests, run them, and make sure everything is green.
There is no need to make some new way of classifying tests. The way we classify tests is that tests that do not work are in Mantis as bug reports. If you want to work on them then you can easily file them in.
-Ralph Johnson
Ralph Johnson wrote:
Failing tests do not belong in the image. They belong in Mantis. If you have a test that ought to run but does not then post it to Mantis.
We should not release an image with failing tests. In general, the 3.10 release team will not accept changes that break tests. If the image contains failing tests then it is hard for people to tell that their proposed changed breaks them. It is crucial that there be NO failing tests in the image so that anyone can easily check whether their change caused any of the tests to fail. Go to the test runner, select all tests, run them, and make sure everything is green.
There is no need to make some new way of classifying tests. The way we classify tests is that tests that do not work are in Mantis as bug reports. If you want to work on them then you can easily file them in.
Is there not a difference between an image under development and a released image? I think so. A released image should not have failing tests but it should also not contain the code that the test failed on. So, defect #5527 should read:
"Some of the tests in the final released 3.9 image fail. You should be able to run all tests in a *RELEASED* image and get a green bar in the TestRunner."
Also, what do you mean by this:
"... and I deleted some of them and moved the rest of them to subclasses in a category FailingTests, which I then deleted. I have attached a change file that makes all the tests green in the final released image."
Did you delete the tests AND the code that it tests? Just removing the tests doesn't really help much if the code that it tests is still there. I'm sure you agree with this, so that is why I'm a bit stumped.
Did you delete the tests AND the code that it tests? Just removing the tests doesn't really help much if the code that it tests is still there. I'm sure you agree with this, so that is why I'm a bit stumped.
You completely miss the point. Leaving broken tests in the image does not cause the tests to be fixed. Broken tests have been in the image for years and they did not help.
There are a lot of bugs in Squeak. Squeak is, in general, of "experimental" quality. It is amazing to me that people actually develop commercial products with it, but they do. I would like to raise the quality of Squeak. Part of this is making sure that there is a regression test suite that actually works, and preventing people from breaking it. There are lots of other parts, of course. The image should be smaller, it should have more tests, more bugs should be fixed, there should not be any unimplemented messages. No one thing will improve quality, but a lot of little things will.
In general, it is not easy to find the code that a test exercises. What you are asking does not make sense.
Please run the tests in 3.9, look at the ones that are broken, and try to figure out what code is bad and should be deleted. It will be an enlightening experience.
-Ralph Johnson
Ralph Johnson wrote:
Did you delete the tests AND the code that it tests? Just removing the tests doesn't really help much if the code that it tests is still there. I'm sure you agree with this, so that is why I'm a bit stumped.
You completely miss the point. Leaving broken tests in the image does not cause the tests to be fixed. Broken tests have been in the image for years and they did not help.
There are a lot of bugs in Squeak. Squeak is, in general, of "experimental" quality. It is amazing to me that people actually develop commercial products with it, but they do. I would like to raise the quality of Squeak. Part of this is making sure that there is a regression test suite that actually works, and preventing people from breaking it. There are lots of other parts, of course. The image should be smaller, it should have more tests, more bugs should be fixed, there should not be any unimplemented messages. No one thing will improve quality, but a lot of little things will.
In general, it is not easy to find the code that a test exercises. What you are asking does not make sense.
Please run the tests in 3.9, look at the ones that are broken, and try to figure out what code is bad and should be deleted. It will be an enlightening experience.
I agree with you that there are a lot of bugs in squeak and they should be fixed. I'm not suggesting that they should be left! I am thankful that you want to raise the quality of squeak, I'm sure everyone does. But, you are taking an active role. Thanks.
Forgive me for missing your point and please bare with me while I ask another question. There's a difference between tests broken and the code broken. You said:
"Leaving broken tests in the image does not cause the tests to be fixed. Broken tests have been in the image for years and they did not help."
So, you are only talking about removing tests that don't work correctly? Not tests that work correctly and fail because the code is buggy.
Ok, that makes sense.
So, you are only talking about removing tests that don't work correctly? Not tests that work correctly and fail because the code is buggy.
If we know that a test is improper then we just delete it. We did some of that. If we know how to fix it then we fix it. We did a little of that. Sometimes it is not clear whether a test is correct. Sometimes we know there is a bug but we don't know how to fix it. In both of those cases, we moved the tests to Mantis.
Moving a test out of the image does not mean it has gone forever. A test that fails is a bug, either a test that shows a bug in the code or a buggy test. Bug reports are supposed to be in Mantis. The tests in the image should all work so we can tell that if we run them after we make a change and some of them fail then we made a mistake.
-Ralph
Ralph Johnson wrote:
...[several excellent points]...
There are a lot of bugs in Squeak. Squeak is, in general, of "experimental" quality. It is amazing to me that people actually develop commercial products with it, but they do. ... [more excellent points]...
Relative to what I'd like to see, there are indeed a lot of bugs in Squeak. While I don't want to take anything away from the thrust of Ralph's plan, I'd like the record to show that that there is not necessarily consensus that Squeak is buggy relative to other platforms - commercial or otherwise.
For example, I spend far too much time on the following technology stack:
A financial application costing tens of $M and responsible for processing several $B each year. PeopleSoft Oracle Windows
Relative to that, Squeak is of absolute perfect quality.
Cheers, -Howard
Hello Howard,
+1
just replace Oracle with other real big players in the software market.
And to Ralph: The plan that running the tests with the knowledge that *every* red is introduced by me is very good!
Herbert mailto:herbertkoenig@gmx.net
Couldn't we remove the failing tests and put them in a sort of "unstable", since presumably they are like a task list of things that we want to work at some point?
There should be no failing tests in a stable release. If you *want* your test to fail, then just reverse the assertion so it is a green test when it works the way it should. And for tests that fail if something is not present, can't the test check if what ever it is actually is present first?
Of course you could keep the failing tests in the stable image if you can put them in a selection like "future work" or something that doesn't show up unless you turn it on. Otherwise deleting sounds ok to me. For the sanity of any release team, red *has to* mean broken.
From: Elod Kironsky kironsky@grisoft.cz Reply-To: The general-purpose Squeak developers listsqueak-dev@lists.squeakfoundation.org To: The general-purpose Squeak developers listsqueak-dev@lists.squeakfoundation.org Subject: Re: improving the quality of the image Date: Mon, 29 Jan 2007 11:47:14 +0100
Philippe Marschall wrote:
2007/1/29, Elod Kironsky kironsky@grisoft.cz:
Philippe Marschall wrote:
2007/1/26, Bert Freudenberg bert@freudenbergs.de:
On Jan 26, 2007, at 16:03 , Philippe Marschall wrote:
2007/1/26, Ralph Johnson johnson@cs.uiuc.edu: > One of my goals for 3.10 is to improve the quality of the image.
Our
> first release (coming soon!) will have only green tests, and each > following release will have only green tests.
How does removing failing tests improve the quality?
Woa, where does that hostility come from? There is another way to ensure all tests are green, besides removing the failing ones.
What hostility? I could not see why this improves the quality because to me the first step to fix a problem is to admit that you have a problem. Failing tests are pointer to problems for me. Removing failing tests because they can not be fixed today or tomorrow looked to me like an attempt to hide hide a problem. So I asked and now I know the reason why it was done.
Philippe
Philippe, where did you read that failing tests will be removed? "First release will have only green tests" means, that all tests remain and will pass, not fail. There will be no test removal at all! I'm, pretty sure you misunderstood something.
http://bugs.impara.de/view.php?id=5527
Philippe
Sorry Philippe, then I have to agree with you and join to Goran's proposition to classify the test, removing them is not a good solution I think.
Elod
_________________________________________________________________ Check out all that glitters with the MSN Entertainment Guide to the Academy Awards® http://movies.msn.com/movies/oscars2007/?icid=ncoscartagline2
when it works the way it should. And for tests that fail if something is not present, can't the test check if what ever it is
So a "to do", or a "known issue" test is a test that fails when the code that works correctly is not present. ;-)
actually is present first?
So it is only a specific instance of a more generalised case
Categorisation handles all of this an more, and at least in the beginning my early implementation used less methods and code than the existing hard coded mechanism.
Keith
___________________________________________________________ All new Yahoo! Mail "The new Interface is stunning in its simplicity and ease of use." - PC Magazine http://uk.docs.yahoo.com/nowyoucan.html
squeak-dev@lists.squeakfoundation.org
-
Bert Freudenberg -
Brad Fuller -
Colin Putney -
Edgar J. De Cleene -
Elod Kironsky -
Göran Krampe -
Herbert König -
Howard Stearns -
J J -
Keith Hodges -
Milan Zimmermann -
nicolas cellier -
Philippe Marschall -
Ralph Johnson -
stephane ducasse