On 6 March 2013 21:37, Jeff Gonis jeff.gonis@gmail.com wrote:
Hi Frank,
So to my chagrin the performance tests on Squeak-CI are pretty much worse than useless, displaying up to a 100% difference in times between builds. For my part I am looking into the SMark framework that Stefan Marr offered and working to learn that and see how I could use it.
But I was curious if there were some other steps I could take in the meantime to try and get the times to be a little more repeatable. The first thing that comes to mind is that we "nice" the vm with I believe the default value before running it for both the tests and the benchmarks, I don't know how to log into the server Jenkins is running on so I can't see what the default value of nice is, but I am assuming it is 0 as that seems fairly standard. Do you think it would be reasonable to try out a high-priority value for at least the benchmarks portion of the SqueakTrunk build. Fire off the benchmarks at a -20 priority and see if we can get some repeatability?
I am not sure about the impacts this might have elsewhere, which is why I wanted to run it by you before sending a pull request. If there is some other easy win that I have overlooked that leaps to mind please let me know, otherwise I will keep plugging away with SMark, and also look into running the tests multiple times to warm up Cog.
Thanks for your help, Jeff G.
Hi Jeff,
The box-admins folk might have further insight, but I think renicing to be hardcore might be the sensible thing. The performance tests don't run for long, and would only run with the SqueakTrunk build, so shouldn't impact too much on things... but I don't recall what other services run on that box.
frank
On 03/06/2013 03:57 PM, Frank Shearar wrote:
On 6 March 2013 21:37, Jeff Gonisjeff.gonis@gmail.com wrote:
Hi Frank,
So to my chagrin the performance tests on Squeak-CI are pretty much worse than useless, displaying up to a 100% difference in times between builds. For my part I am looking into the SMark framework that Stefan Marr offered and working to learn that and see how I could use it.
But I was curious if there were some other steps I could take in the meantime to try and get the times to be a little more repeatable. The first thing that comes to mind is that we "nice" the vm with I believe the default value before running it for both the tests and the benchmarks, I don't know how to log into the server Jenkins is running on so I can't see what the default value of nice is, but I am assuming it is 0 as that seems fairly standard. Do you think it would be reasonable to try out a high-priority value for at least the benchmarks portion of the SqueakTrunk build. Fire off the benchmarks at a -20 priority and see if we can get some repeatability?
I am not sure about the impacts this might have elsewhere, which is why I wanted to run it by you before sending a pull request. If there is some other easy win that I have overlooked that leaps to mind please let me know, otherwise I will keep plugging away with SMark, and also look into running the tests multiple times to warm up Cog.
Thanks for your help, Jeff G.
Hi Jeff,
The box-admins folk might have further insight, but I think renicing to be hardcore might be the sensible thing. The performance tests don't run for long, and would only run with the SqueakTrunk build, so shouldn't impact too much on things... but I don't recall what other services run on that box.
frank
Please define clearly "don't run for long". At the moment only Jenkins is running on that server so in the short run at most, it would impact Jenkins. However it the not too distant future the mailing lists will be on that server and while a little delay is no big deal I would like some idea how much before saying "OK".
Ken
On 7 March 2013 02:52, Ken Causey ken@kencausey.com wrote:
On 03/06/2013 03:57 PM, Frank Shearar wrote:
On 6 March 2013 21:37, Jeff Gonisjeff.gonis@gmail.com wrote:
Hi Frank,
So to my chagrin the performance tests on Squeak-CI are pretty much worse than useless, displaying up to a 100% difference in times between builds. For my part I am looking into the SMark framework that Stefan Marr offered and working to learn that and see how I could use it.
But I was curious if there were some other steps I could take in the meantime to try and get the times to be a little more repeatable. The first thing that comes to mind is that we "nice" the vm with I believe the default value before running it for both the tests and the benchmarks, I don't know how to log into the server Jenkins is running on so I can't see what the default value of nice is, but I am assuming it is 0 as that seems fairly standard. Do you think it would be reasonable to try out a high-priority value for at least the benchmarks portion of the SqueakTrunk build. Fire off the benchmarks at a -20 priority and see if we can get some repeatability?
I am not sure about the impacts this might have elsewhere, which is why I wanted to run it by you before sending a pull request. If there is some other easy win that I have overlooked that leaps to mind please let me know, otherwise I will keep plugging away with SMark, and also look into running the tests multiple times to warm up Cog.
Thanks for your help, Jeff G.
Hi Jeff,
The box-admins folk might have further insight, but I think renicing to be hardcore might be the sensible thing. The performance tests don't run for long, and would only run with the SqueakTrunk build, so shouldn't impact too much on things... but I don't recall what other services run on that box.
frank
Please define clearly "don't run for long". At the moment only Jenkins is running on that server so in the short run at most, it would impact Jenkins. However it the not too distant future the mailing lists will be on that server and while a little delay is no big deal I would like some idea how much before saying "OK".
I don't have a trend, but the last build spent 38 seconds running the performance tests. Compared to the time taken to possibly compile a VM, update an image, and run the tests, the benchmarking's peanuts, at least for now.
frank
Ken
On 03/07/2013 01:26 AM, Frank Shearar wrote:
On 7 March 2013 02:52, Ken Causeyken@kencausey.com wrote:
On 03/06/2013 03:57 PM, Frank Shearar wrote:
On 6 March 2013 21:37, Jeff Gonisjeff.gonis@gmail.com wrote:
Hi Frank,
So to my chagrin the performance tests on Squeak-CI are pretty much worse than useless, displaying up to a 100% difference in times between builds. For my part I am looking into the SMark framework that Stefan Marr offered and working to learn that and see how I could use it.
But I was curious if there were some other steps I could take in the meantime to try and get the times to be a little more repeatable. The first thing that comes to mind is that we "nice" the vm with I believe the default value before running it for both the tests and the benchmarks, I don't know how to log into the server Jenkins is running on so I can't see what the default value of nice is, but I am assuming it is 0 as that seems fairly standard. Do you think it would be reasonable to try out a high-priority value for at least the benchmarks portion of the SqueakTrunk build. Fire off the benchmarks at a -20 priority and see if we can get some repeatability?
I am not sure about the impacts this might have elsewhere, which is why I wanted to run it by you before sending a pull request. If there is some other easy win that I have overlooked that leaps to mind please let me know, otherwise I will keep plugging away with SMark, and also look into running the tests multiple times to warm up Cog.
Thanks for your help, Jeff G.
Hi Jeff,
The box-admins folk might have further insight, but I think renicing to be hardcore might be the sensible thing. The performance tests don't run for long, and would only run with the SqueakTrunk build, so shouldn't impact too much on things... but I don't recall what other services run on that box.
frank
Please define clearly "don't run for long". At the moment only Jenkins is running on that server so in the short run at most, it would impact Jenkins. However it the not too distant future the mailing lists will be on that server and while a little delay is no big deal I would like some idea how much before saying "OK".
I don't have a trend, but the last build spent 38 seconds running the performance tests. Compared to the time taken to possibly compile a VM, update an image, and run the tests, the benchmarking's peanuts, at least for now.
frank
OK, I guess probably anything up to 5-10 minutes of 'very busy' time is probably OK. I'm assuming this is a once a day task. As it gets longer we may just want to consider scheduling it at otherwise 'quiet' times.
Ken
On 07 Mar 2013, at 7:48, Ken Causey ken@kencausey.com wrote:
On 03/07/2013 01:26 AM, Frank Shearar wrote:
On 7 March 2013 02:52, Ken Causeyken@kencausey.com wrote:
On 03/06/2013 03:57 PM, Frank Shearar wrote:
On 6 March 2013 21:37, Jeff Gonisjeff.gonis@gmail.com wrote:
Hi Frank,
So to my chagrin the performance tests on Squeak-CI are pretty much worse than useless, displaying up to a 100% difference in times between builds. For my part I am looking into the SMark framework that Stefan Marr offered and working to learn that and see how I could use it.
But I was curious if there were some other steps I could take in the meantime to try and get the times to be a little more repeatable. The first thing that comes to mind is that we "nice" the vm with I believe the default value before running it for both the tests and the benchmarks, I don't know how to log into the server Jenkins is running on so I can't see what the default value of nice is, but I am assuming it is 0 as that seems fairly standard. Do you think it would be reasonable to try out a high-priority value for at least the benchmarks portion of the SqueakTrunk build. Fire off the benchmarks at a -20 priority and see if we can get some repeatability?
I am not sure about the impacts this might have elsewhere, which is why I wanted to run it by you before sending a pull request. If there is some other easy win that I have overlooked that leaps to mind please let me know, otherwise I will keep plugging away with SMark, and also look into running the tests multiple times to warm up Cog.
Thanks for your help, Jeff G.
Hi Jeff,
The box-admins folk might have further insight, but I think renicing to be hardcore might be the sensible thing. The performance tests don't run for long, and would only run with the SqueakTrunk build, so shouldn't impact too much on things... but I don't recall what other services run on that box.
frank
Please define clearly "don't run for long". At the moment only Jenkins is running on that server so in the short run at most, it would impact Jenkins. However it the not too distant future the mailing lists will be on that server and while a little delay is no big deal I would like some idea how much before saying "OK".
I don't have a trend, but the last build spent 38 seconds running the performance tests. Compared to the time taken to possibly compile a VM, update an image, and run the tests, the benchmarking's peanuts, at least for now.
frank
OK, I guess probably anything up to 5-10 minutes of 'very busy' time is probably OK. I'm assuming this is a once a day task. As it gets longer we may just want to consider scheduling it at otherwise 'quiet' times.
Ken
Levente's script polls every 15 mins so at most this could run something like 24*3 times in a day. But typically the commits occur in chunks, so I reckon 10 times in a day would be a wonderful problem to have.
frank
On 7 March 2013 08:21, Frank Shearar frank.shearar@gmail.com wrote:
On 07 Mar 2013, at 7:48, Ken Causey ken@kencausey.com wrote:
On 03/07/2013 01:26 AM, Frank Shearar wrote:
On 7 March 2013 02:52, Ken Causeyken@kencausey.com wrote:
On 03/06/2013 03:57 PM, Frank Shearar wrote:
On 6 March 2013 21:37, Jeff Gonisjeff.gonis@gmail.com wrote:
Hi Frank,
So to my chagrin the performance tests on Squeak-CI are pretty much worse than useless, displaying up to a 100% difference in times between builds. For my part I am looking into the SMark framework that Stefan Marr offered and working to learn that and see how I could use it.
But I was curious if there were some other steps I could take in the meantime to try and get the times to be a little more repeatable. The first thing that comes to mind is that we "nice" the vm with I believe the default value before running it for both the tests and the benchmarks, I don't know how to log into the server Jenkins is running on so I can't see what the default value of nice is, but I am assuming it is 0 as that seems fairly standard. Do you think it would be reasonable to try out a high-priority value for at least the benchmarks portion of the SqueakTrunk build. Fire off the benchmarks at a -20 priority and see if we can get some repeatability?
I am not sure about the impacts this might have elsewhere, which is why I wanted to run it by you before sending a pull request. If there is some other easy win that I have overlooked that leaps to mind please let me know, otherwise I will keep plugging away with SMark, and also look into running the tests multiple times to warm up Cog.
Thanks for your help, Jeff G.
Hi Jeff,
The box-admins folk might have further insight, but I think renicing to be hardcore might be the sensible thing. The performance tests don't run for long, and would only run with the SqueakTrunk build, so shouldn't impact too much on things... but I don't recall what other services run on that box.
frank
Please define clearly "don't run for long". At the moment only Jenkins is running on that server so in the short run at most, it would impact Jenkins. However it the not too distant future the mailing lists will be on that server and while a little delay is no big deal I would like some idea how much before saying "OK".
I don't have a trend, but the last build spent 38 seconds running the performance tests. Compared to the time taken to possibly compile a VM, update an image, and run the tests, the benchmarking's peanuts, at least for now.
frank
OK, I guess probably anything up to 5-10 minutes of 'very busy' time is probably OK. I'm assuming this is a once a day task. As it gets longer we may just want to consider scheduling it at otherwise 'quiet' times.
Ken
Levente's script polls every 15 mins so at most this could run something like 24*3 times in a day. But typically the commits occur in chunks, so I reckon 10 times in a day would be a wonderful problem to have.
Apparently I need a new prescription for my glasses: the benchmarks take 2.5 minutes to run, not 38 seconds.
frank
box-admins@lists.squeakfoundation.org