Thanks for your reply, Igor! Inline.
On May 3, 2011, at 7:44 PM, Igor Stasenko siguctua@gmail.com wrote:
Thank you, Casey for willing to help. Hudson status: up and working. And most of the times it does its job. Except that we would like a more finer control on it.
Finer control of what tests get run? Out of curiosity, are you just building the VM with Hudson, or are you also running SUnit tests there? Has anyone done the work to make SUnit output JUnit XML reports (needed for Hudson to do detailed reporting?)
I'm more focused on doing image level CI, personally, because I want to know right away when I break something in the image, and because I'm not doing much VM hacking (yet.)
Currently Hudson are set up for us by INRIA engineers, and while it was good from the beginning (we don't need to spend time setting up things by ourselves), today our demands looks like outgrew what admins at INRIA could propose :)
How so? What are the actual problems? I solve people/process problems like these professionally, in addition to grinding out test automation. For you people, for the love you've shown, I'll do it for free;) Let me at em!
We will have a meeting on this week and discuss our problems.
Can I come? Seriously, I live for solving this problem. Please invite me if you can: I promise to be on my very best behavior.
We're certainly want to have more open access to servers by community members, so they could set up, configure and run their own jobs.
That's a great idea, if people in the community are lining up to do so.
Stef says that he want servers to burn in flames (means busy with always building something).
Good thing you qualified that, because in vernacular American English, that means he wishes someone would actually set the boxes on fire:P
I agree though. A CI box should always be hot! As in, put another fan on that thing: it's too hot.
But we meet some organizational/political problems we need to overcome first.
Would you be willing develop that statement?
I think Stephane could say more about it.
CC'ing him then. Hi Stephane! I found some free time and I want to help the Pharo and Squeak communities with their CI efforts, because CI is love. Can you help me understand any of the organizational/political challenges Igor speaks of? I'm also interested in technical stumbling blocks that I might be able to help with.
What help i personally might need? Well.. building VMs on many various platforms and making working cmake configs for them. So, mainly its about creating and testing new cmake configurations.
Ah, I see. I mostly want to do CI on the image(s.) I'm *really* glad you guys are running tests against the VM, but the VM is currently mostly outside of my area of focus. I want to know right away if my code is going to break the Trunk; this is my main motivator -- er, that and did I mention -- CI is love!
Oh.. yeah.. we would like to set up a benchmarking suites for running benchmarks after each VM build. And then outputting results to some file(s), so we can detect performance regression from build to build etc etc.
Also really cool.
That would be really nice. But that means some coding, because first we need to create some kind of benchmarking suite, and also some layer to output results, which hudson can understand (expect heavy xml resistance ;)
If you're using Hudson, you want to output JUnit XML. There isn't a better way to do it without modifying Hudson.
Except... Well, you could write a plugin (yuck) or better yet...
What's the strongest resistance? Would people rather hack a lot of Java code than write a little bit of Smalltalk (or Perl or whatever in the case of the VM) to get XML output? Cause I'd rather do the latter.
If you guys *hate* XML so much (I hate it almost that much) that you'd rather do more work than less in order to avoid dealing with it, before you go, you might look at TAP.
TAP stands for Test Anything Protocol, and originated with Perl's Test::More (to my knowledge anyway -- Randal? Historian hat?)
The reason I'm telling you this is, the reporting output format is ridiculously simple to implement, and IIRC there's a plugin for Hudson that'll let you use TAP output instead of JUnit XML. You don't need Perl to use TAP output because the format is really just a dirt simple convention for reporting on test results.
This is what we did to deal with the fact that we had a bunch of different test runners for different languages with different output formats and wanted consistent reporting when I was working on the problem at WhitePages.com, we integrated TAP output support for the few languages that didn't just have a library that did it.
At WP, we were using a different CI server (CruiseControl.rb) and at that point in time, we had to modify CruiseControl in order to make it understand TAP output. In the case of Hudson, as I stated, I'm pretty sure there's a plugin that does it for you.
Just a thought.
-- Best regards, Igor Stasenko AKA sig.