When putting together a new buildbot, or when trying to understand a build problem better, you potentially want to test it locally. Thanks to people from the community this has become *tremendously* simpler not too long ago. When I started to maintain the buildbot/fleet of buildbots as part of my day job, this was unthinkable. So, I said it during my FOSDEM presentation, and I say it again: Thank you!
In this post I’m going to go over the two different scenarios that I face every now and then: (1) reproducing an existing buildbot locally, and (2) developing a new buildbot locally. The first case is reasonable to know, in case you want or need to check if a certain issue reproduces locally. The latter case is the more interesting in my opinion since it enables you to develop a new buildbot from scratch. This is especially helpful when you make larger changes or start from scratch. In my case, as an example, we are in the process to switch our buildbot fleet from so-called OpenMPBuilder-based buildbots to AnnotatedBuilder buildbots. The difference is mainly the flexbility in the latter, vs more off-the-shelve functionality in the former.
However, I’ll first go over the easier use case to reproduce an existing one locally and then explain the case to develop a new builder from scratch.
Setting Up Local Test Instance To Reproduce
To reproduce a buildbot locally, the community landed several changes that make it much simpler. The machine that I was working on this runs AlmaLinux 9, if you run a different distribution, things may be a little different. The documentation shows all steps needed. I mostly want to add a few notes here.
Change IP address server binds to: The address that the test instance binds to is configured to be 127.0.0.1:<port>. That is fine when you develop / test on the same machine, however, if you want to access the buildbot test instance from another machine, you need to change that to the IP address of the machine in the local network. Otherwise, the buildbot server UI won’t show up. Also, somewhat obviously, you need to configure a potentially existing firewall to allow incoming connections on that IP address + port combination. The interface is configured in the file buildbot/osuosl/master/master.cfg. What exactly to put there depends on your local setup.
Note! Config comes from llvm-zorg: The configuration of the actual builders / workers is pulled from the llvm-zorg repository. This means that in the setup outlined in the documentation, you can really just reproduce an existing builder w/o local modifications or similar. In order to do the latter some more changes are required. Those changes are covered in the next section.
Setting Up Local Test Instance For Development
When you want to develop a new builder or try fixes to an existing configuration, you need some more changes. This is also mentioned in the upstream documentation. Those changes may be different, depending on the situation. One thing that is worth pointing out: I think you should fork llvm-zorg and work only in local working copy from that fork while developing on a new builder. That is mostly to not confuse environments and remotes.
I’ll cover here what I did while developing a new builder. The changes were still pretty simple, however, it took me longer to remember what I did than I had hoped. So, I thought it makes sense to write them down here. All the changes I outline now must be done before starting the buildbot test instance server. Only at the very end of this post you should actually start the server.
The goal: Create and locally run a new buildbot that is based on an AnnotatedBuilder and runs tests in the llvm-test-suite.
Let’s assume we have already finalized the contribution to llvm-test-suite and know exactly which commands we actually want to run in the buildbot. For an AnnotatedBuilder, one has to define the script that it eventually executes and which has all the steps. This gives it the most flexibility and (to us most important) allows to easily and fast change the actual buildbot once it is in production, as opposed to a required main-server restart to propagate config changes. However, the script lives inside the llvm-zorg repository and, without modification, the local test instance will pull from llvm-zorg. This means that you would need to commit every change to the community project in order to test it. I personally believe that is not what you should be doing, but work locally until all works and then open PRs to upstream the changes.
In your local clone, you create your normal feature branch that you work on during developing the new builder and push the changes to your fork on the github remote (or wherever, but that is what I do). This gives you a remote from which the buildbot is able to pull changes. Now, we need to tell the buildbot from where it should pull the updates.
Change the way in which the factory creates the builder in: zorg/buildbot/builders/AnnotatedBuilder.py. As an example, while I was developing the buildbot that runs upstream LLVM on the Kokkos library with the HIP backend enabled, I had this diff in my llvm-zorg working copy
diff --git a/zorg/buildbot/builders/AnnotatedBuilder.py b/zorg/buildbot/builders/AnnotatedBuilder.py
index 0b94049b..846ff46d 100644
--- a/zorg/buildbot/builders/AnnotatedBuilder.py
+++ b/zorg/buildbot/builders/AnnotatedBuilder.py
@@ -79,7 +79,11 @@ def getAnnotatedBuildFactory(
name='update-annotated-scripts',
project='zorg',
src_dir='llvm-zorg',
- alwaysUseLatest=True)
+ alwaysUseLatest=True,
+ repourl="https://github.com/jplehr/llvm-zorg.git",
+ branch='feat/hip-tpl'
+ )
if checkout_llvm_sources:
f.addGetSourcecodeSteps()
This change points the local buildbot server test instance to my fork of llvm-zorg on github to pull changes from via the repourl argument. It also sets it to pull from a specific feature branch via the branch argument. When you start the test server after you made these modifications, all AnnotatedBuilder-based buildbots will now pull the llvm-zorg changes from your fork instead of the upstream llvm-zorg repository. That means that you can start developing your annotated script that the worker will use for its builds on the feature branch, push to your fork and the local test instance will pick up the changes with the next build through its automated “update” step.
Finally, you create the entry inside the workers.py to define a worker and inside the builders.py file to define your new buildbot, e.g., such as done here. You can then start the local test instance and start the local test worker. Then go to the local web interface view and see your new builder online and working. You can monitor the status through the web UI as normal. When you see something to fix, all you need to do is implement the fix in the annotated script, commit and push to feature branch on your llvm-zorg fork and the changes will be picked-up by the test instance with the next “update” step.
I hope this little post helps people (or at least me in the future) when they need to go through these motions. If you like it or spot an error, consider dropping a comment or letting me know through discord or whatever.
No responses yet