I have some performance test result printed in gitlab job console like below,
Actual ResultSet Total Records [260]!
Performance Test Successfully Completed! Total time [1354108]
How to generate a performance report out of which and show in gitlab pipeline?
Related
Current Scenario: We have 5 jobs in RT executions and each job is running on 5 different agents. So we have 5 cucumber reports.
Expected Scenario: We need a consolidated Cucumber HTML Report of all 5 jobs.
How it can be resolved? We are working in azure Devops.
The Cucumber Html report is a single page html application. If you open it up in an editor you'll see that it contains a an array of messages.
These messages can also be obtained by using the messages:target/messages.ndjson plugin. If you can obtain in a single job the messages from each parallel job you can then merge these and pass them to the html-formatter.
To merge the files you may have to filter out some messages. Definitively requires writing some code, don't think anyone has done this before, but shouldn't be impossible.
Consider a pipeline with the following jobs:
build: Runs the build and takes 1 minute
report: Runs a static code analysis, posts the result to the MR and takes 59 minutes
Developers should be informed about the results of the report stage as soon as possible, but it should not block the MR from being merged. The pipeline should behave like this:
build must always be successful, before the MR can be merged.
report should always be started eventually and executed successfully, but it should not be mandatory to wait for it in order to be able to merge the MR.
Is there a possibility in gitlab to create such a pipeline?
So far, I am aware of the following options:
Disable the "Pipelines must succeed" setting: In this case, the MR can be merged, even if build is not successful.
Set allow_failure for report to true. In this case, The MR can be merged after build has completed by cancelling the report job, but this violates the requirement that the report should always be executed. Also it is poor developer experience if you have to cancel an optional job before being able to merge.
Execute the report job after merge. This has two drawbacks:
I will get the report only when the MR is merged instead of as soon as possible.
The report job cannot post it's result to the MR, which would notify the involved persons.
You can move your report job into a child pipeline (= a separate .yml file in your project) and trigger it with the trigger keyword and without strategy: depend.
This allows you to trigger the job without waiting for it and without considering its state in your pipeline.
We set up junit test reporting in gitlab and can see the results in the pipeline.
Is it possible in any way (maybe via API?) to extract a statistic of how often a test fails? For example:
TestX 0/20 times successful
TestY 17/20 times successful
TestZ 19/20 times successful
...
Background: we have very many integration tests and some of them show timing issues which cause them to fail. I would like to identify the tests which fail most often.
In the "Details for Job n section, the UI shows "Status: SUCCEEDED", however one of the Stages shows 22029 succeeded Tasks out of 59400 total tasks. I'm running this through a Python Jupyter notebook running Spark 3.0.1, and I haven't stopped the Spark context yet so the application is still running. In fact, the Stages Tab shows the stage in question as still active. I don't understand how the stage could still be active, yet the Job is listed as Completed and Successful in the UI.
The relevant code (I think) is below, where I try to parallelize as much as possible many SQL queries and then union the result dataframes together. Lastly, I'm writing them to cloud storage in parquet.
EDIT: I also can see the same information from the REST API using the endpoints documented here in the docs, and those values are the same as I see in the Web UI.
There are no jobs appearing in the Jobs tab as failed, and I believe that ultimately the data is successfully written and correct.
I have seen in the logs many instances of Dropping event from queue appStatus. This likely means one of the listeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler. Because of that, I am experimenting with the parameter spark.scheduler.listenerbus.eventqueue.capacity to increase it and see if that results in a difference in the reporting of Succeeded and Total tasks for that stage.
Upon increasing spark.scheduler.listenerbus.eventqueue.capacity from default of 10000 to 65000, there seems to be a corresponding decrease in events dropped, as well as an increase in Succeeded Tasks reported for that stage, improving to ~ 47K from ~ 22K. I have also noticed that the difference in Succeeded and Total tasks for that stage is on the order of the number of dropped events in the log so I will see if limiting the dropped events can resolve the discrepancy.
def make_df(query: str):
df = spark.sql(query)
return df
spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
df_list = list(map(make_df, queries))
df = functools.reduce(lambda x, y: x.union(y), df_list)
df.repartition("col1", "col2")\
.write.partitionBy("col1", "col2")\
.mode("overwrite")\
.parquet(path)
Why would my job be reporting as Successful when there are still tasks remaining that aren't successful?
Using automation testing running Cucumber through Jenkins, I am able to get the results for execution, however I would need to accumulate the total results, any idea or tool that could allow me to aggregate the results by day, week, month,...