I would like to determine which schedulers to trigger depending on the branch name, from inside the build factory - if that's possible.
Essentially I have a builder that is doing all the common build steps to compile package etc, and then has a bunch of trigger steps that trigger a bunch of tests (via triggerable schedulers).
However, I would like to configure the type of tests that get started (eg which schedulers are triggered) to depend on the branch name. So far I've tried to add the change_filter arg to my Triggerable scheduler, but it seems that it doesn't accept that argument. I guess that makes sense because it supposed to be Triggered, so maybe it doesn't care about using a change filter. That seems a bit strange though because Dependent schedulers do accept this kwarg.
So far the correct way to set this up is not clear to me.
I guess my questions are really:
Is there a way to use renderables / properties to decide which schedulers to trigger (based on the branch name for example)?
Is there a better way to do this? Perhaps create separate schedulers for the build that apply the change filter I need and have a build factory that triggers the correct tests, but that's not very DRY.
I came back to leave this here in case it might help someone with a tricky buildbot setup.
I solved this by making all of the dependent schedulers (for specific types of tests) into triggerable schedulers. Then I created main build schedulers for each subset of tests, each with a change filter and regex for the branches that should undergo that subset of tests. Finally, I created the buildfactory for each main scheduler by passing it only the triggerable schedulers for the test that that specific type of main scheduler should run.
For my current use case, this works great!
Related
I’ve been tasked with investigating whether something can be added to our MR approval process to highlight whether proposed Merge Requests (MRs) in GitLab, which are C++ based code, contain any new unit tests or modifications to existing tests. The overall aim is to remind developers and approvers that they need to consider unit testing.
Ideally a small script would run and detect the presence of additional tests or changes (this I can easily write, and I accept that there’s a limit to how much can be done here) and display a warning on the MR if they weren’t detected.
An addition step, if at all possible, would be to block the MR until either, further commits were pushed that meet the criteria, or an (extra/custom) GitLab MR field is completed explaining why unit testing is not appropriate for this change. This field would be held with the MR for audit purposes. I accept that this is not foolproof but am hoping to pilot this as part of a bigger push for more unit test coverage.
As mentioned, I can easily write a script in, say, Python to check for unit tests in the commit(s), but what I don’t know is whether/how I can hook this into the GitLab MR process (I looked at web-hooks but they seem to focus on notifying other systems rather than being transactional) and whether GitLab is extensible enough for us to achieve the additional step above. Any thoughts? Can this be done, and if so, how would I go about it?
measuring the lack of unit tests
detect the presence of additional tests or changes
i think you are looking for the wrong thing here.
the fact that tests have changed or that there are any additional tests does not mean, that the MR contains any unit tests for the submitted code.
the underlying problem is of course a hard one.
a good approximation of what you want is typically to check how many lines of code are covered by the test-suite.
if the testsuite tests more LOCs after the MR than before, then the developer has done their homework and the testsuite has improved. if the coverage has grown smaller, than there is a problem.
of course it's still possible for a user to submit unit tests that are totally unrelated to their code changes, but at least the overall coverage has improved (or: if you already have a 100% coverage before the MR, then any MR that keeps the coverage at 100% and adds new code has obviously added unit tests for the new code).
finally, to come to your question
yes, it's possible to configure a gitlab-project to report the test-coverage change introduced by an MR.
https://docs.gitlab.com/ee/ci/pipelines/settings.html#test-coverage-parsing
you obviously need to create a coverage report from your unittest run.
how you do this depends on the unittesting framework you are using, but the gitlab documentation gives some hints.
You don't need a web hook or anything like that. This should be something you can more or less trivially solve with just an extra job in your .gitlab-ci.yml. Run your python script and have it exit nonzero if there are no new tests, ideally with an error message indicating that new tests are required. Now when MRs are posted your job will run and if there are no new tests the pipeline will fail.
If you want the pipeline to fail very fast, you can put this new job at the head of the pipeline so that nothing else runs if this one fails.
You will probably want to make it conditional so that it only runs as part of an MR, otherwise you might get false failures (e.g. if just running the pipeline against some arbitrary commit on a branch).
In short:
Find a way to do simple parsing of a variable that is already available to environment:url as part of a pipeline job.
The particulars: A job that is triggered by a git tag following a git-flow release finish, such as v1.32.7. GitLab makes this available in CI_BUILD_TAG. What I would like is to to be able to only use the major version part, e.g. v1 in environment:name and environment:url.
Does anyone have a clever way of solving this? I've considered maybe having hooks that insert this value into the code itself, but I'm curious as to what solutions others have arrived at.
I'm aware that GitLab has strict limitations as to where variables can be expanded, and which variables can be used when. Here is an overview: https://docs.gitlab.com/ee/ci/variables/where_variables_can_be_used.html#gitlab-internal-variable-expansion-mechanism
My question is regarding the use of data that is already available to GitLab, such as the variables in question. And not the act of transferring data evaluated by runners to GitLab, something that is fairly problematic given the architecture, and otherwise discussed here:
https://gitlab.com/gitlab-org/gitlab-ce/issues/27921
https://gitlab.com/gitlab-org/gitlab-ce/issues/28314
Unfortunately, that can't be done strictly within a .gitlab-ci.yml file. It's a technical limitation. The environment name and url are evaluated on the GitLab (server) side, not by the runners. So, there's no access to variables declared in the script block, as you've already discovered. I've yet to find any clever workarounds.
One possibility, related to your comment about “hooks that insert this value into the code itself,” is to have something call the GitLab API to run a pipeline for a tag that parses out the v1 part and passes it in as a variable. Pipeline variables can be used for environment names and urls.
https://docs.gitlab.com/ee/api/pipelines.html#create-a-new-pipeline
I am automating acceptance tests defined in a specification written in Gherkin using Elixir. One way to do this is an ExUnit addon called Cabbage.
Now ExUnit seems to provide a setup hook which runs before any single test and a setup_all hook, which runs before the whole suite.
Now when I try to isolate my Gherkin scenarios by resetting the persistence within the setup hook, it seems that the persistence is purged before each step definition is executed. But one scenario in Gherkin almost always needs multiple steps which build up the test environment and execute the test in a fixed order.
The other option, the setup_all hook, on the other hand, resets the persistence once per feature file. But a feature file in Gherkin almost always includes multiple scenarios, which should ideally be fully isolated from each other.
So the aforementioned hooks seem to allow me to isolate single steps (which I consider pointless) and whole feature files (which is far from optimal).
Is there any way to isolate each scenario instead?
First of all, there are alternatives, for example: whitebread.
If all your features, needs some similar initial step, maybe background steps are something to look into. Sadly those changes were mixed in a much larger rewrite of the library that newer got merged into. There is another PR which also is mixed in with other functionality and currently is waiting on companion library update. So currently that doesn't work.
Haven't tested how the library is behaving with setup hooks, but setup_all should work fine.
There is such a thing as tags. Which I think haven't yet been published with the new release, but is in master. They work with callback tag. You can look closer at the example in tests.
There currently is a little bit of mess. I don't have as much time for this library as I would like to.
Hope this helps you a little bit :)
I would like to execute extra actions after a successful fork .
(e.g. automate the creation of a CruiseControl.net project).
Which code should I modify? Where should I start from?
There are many options to implement that, according to the source code.
the first option should be: modify app_services/projects/fork_service.rb
another option could be: implement a project_service (e.g. model, worker), which will be binded to an external (unexisting) API which will manage the complexity of the project creation
(more links when my reputation will be high enough ;-))
I'm writing a feature in cucumber that could be applied to a number of objects that can be programmaticaly determined. Specifically, I'm writing a smoke test for a cloud deployment (though the problem is with cucumber, not the cloud tools, thus stack overflow).
Given a node matching "role:foo"
When I connect to "automatic.eucalyptus.public_ipv4" on port "default.foo.port"
Then I should see "Hello"
The given does a search for nodes with the role foo does and the automatic.eucalyptus... And port come from the node found. This works just fine... for one node.
The search could retun multiple nodes in different environments. Dev will probably return one, test and integration a couple, and prod can vary. The given already finds all of them.
Looping over the nodes in each step doesn't really work. If any one failed in the When, the whole thing would fail. I've looked at scenarios and cucumber-iterate, but both seem to assume that all scenarios are predefined rather than programmatically looked up.
I'm a cuke noob, so I'm probably missing something. Any thoughts?
Edit
I'm "resolving" the problem by flipping the scenario. I'm trying to integrate into a larger cluster definition language to define repeatedly call the feature by passing the info as an environment variable.
I apologize in advance that I can't tell you exactly "how" to do it, but a friend of mine solved a similar problem using a somewhat unorthodox technique. He runs scenarios that write out scenarios to be run later. The gem he wrote to do this is called cukewriter. He describes how to use it in pretty good detail on the github page for the gem. I hope this will work for you, too.