I'm having an issue with the built in gitlab dast scanning and authentication in the pipeline.
The application that is attempting to be scanned is an angular app using the aspnetzero framework.
In gitlab the cicd file uses the dast UI configuration to setup the job and in the cicd yml file the job spec looks like:
# Include the DAST template
include:
- template: DAST.gitlab-ci.yml
# Your selected site and scanner profiles:
dast:
stage: dast
dast_configuration:
site_profile: "auth"
scanner_profile: "default"
In the site profile the proper data is setup for authentication and then running the dast scanning job, I get an error in the logs like
2022-07-12T22:00:16.000 INF NAVDB Load URL added to crawl graph
2022-07-12T22:00:16.000 INF AUTH Attempting to authenticate
2022-07-12T22:00:16.000 INF AUTH Loading login page LoginURL=https://example.com/account
2022-07-12T22:00:23.000 WRN BROWS response body exceeds allowed size allowed_size_bytes=10000000 request_id=interception-job-4.0 response_size_bytes=11100508 url=https://example.com/main.f3808aecbe8d4efb.js
2022-07-12T22:00:38.000 WRN CONTA request failed, attempting to continue scan error=net::ERR_BLOCKED_BY_RESPONSE index=0 requestID=176.5 url=https://example.com/main.f3808aecbe8d4efb.js
2022-07-12T22:00:39.000 INF AUTH Writing authentication report path=/zap/wrk/gl-dast-debug-auth-report.html
2022-07-12T22:00:39.000 INF AUTH skipping writing of JSON cookie report as there are no cookies to write
2022-07-12T22:00:40.000 FTL MAIN Authentication failed: failed to load login page: expected to find a single element for selector css:#manual_login to follow path to login form, found 0
2022-07-12 22:00:40,059 Browserker completed with exit code 1
2022-07-12 22:00:40,060 BrowserkerError: Failure while running Browserker 1.Exiting scan
sion.ExtensionLoader - Initializing Provides the foundation for concrete message types (for example, HTTP, WebSockets) expose fuzzer implementations.
[zap_server] 13499 [ZAP-daemon] INFO org.parosproxy.paros.extension.ExtensionLoader - Initializing Allows to fuzz HTTP messages.
It seems like container that is doing the dast scanning can't properly load the angular javascript file since it exceeds the allowed response size, and the actual login form does not load. Is there a way to increase the allowed size for the request so that we can have the login form properly load.
I've tried various options like setting the stability timeout variables, and even increasing the memory for the ZAP process (DAST_ZAP_CLI_OPTIONS: '-Xmx3072m' ). but am still getting the same result in that the login form isn't loading, most likely because the javascript isn't loading properly.
The fix looks like to be a gitlab/dast cicd variable issue that isn't in any of the current documentation that I could find.
In order to view all the options or parameters available I update the cicd file with the following:
include:
template: DAST.gitlab-ci.yml
dast:
script:
- /analyze --help
so I could see the options available. From this I was able to find DAST_BROWSER_MAX_RESPONSE_SIZE_MB variable to use. Setting that variable fixed my issue
Related
I use Apache Airflow for daily ETL jobs. I installed it in Azure Kubernetes Service using the provided Helm chart. It's been running fine for half a year, but since recently I'm unable to access the logs in the webserver (this used to always work fine).
I'm getting the following error:
*** Log file does not exist: /opt/airflow/logs/dag_id=analytics_etl/run_id=manual__2022-09-26T09:25:50.010763+00:00/task_id=copy_device_table/attempt=18.log
*** Fetching from: http://airflow-worker-0.airflow-worker.default.svc.cluster.local:8793/dag_id=analytics_etl/run_id=manual__2022-09-26T09:25:50.010763+00:00/task_id=copy_device_table/attempt=18.log
*** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers and workers) have the same 'secret_key' configured in 'webserver' section and time is synchronized on all your machines (for example with ntpd) !!!!!
****** See more at https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key
****** Failed to fetch log file from worker. Client error '403 FORBIDDEN' for url 'http://airflow-worker-0.airflow-worker.default.svc.cluster.local:8793/dag_id=analytics_etl/run_id=manual__2022-09-26T09:25:50.010763+00:00/task_id=copy_device_table/attempt=18.log'
For more information check: https://httpstatuses.com/403
What have I tried:
I've made sure that the log file exists (I can exec into the airflow-worker-0 pod and read the file on command line in the location specified in the error).
I've rolled back my deployment to an earlier commit from when I know for sure it was still working, but it made no difference.
I was using webserverSecretKeySecretName in the values.yaml configuration. I changed the secret to which that name was pointing (deleted it and created a new one, as described here: https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#webserver-secret-key) but it didn't work (no difference, same error).
I changed the config to use a webserverSecretKey instead (in plain text), no difference.
My thoughts/observations:
The error states that the log file doesn't exist, but that's not true. It probably just can't access it.
The time is the same in all pods (I double checked be exec-ing into them and typing date in the command line)
The webserver secret is the same in the worker, the scheduler, and the webserver (I double checked by exec-ing into them and finding the corresponding env variable)
Any ideas?
Turns out this was a known bug with the latest release (2.4.0) of the official Airflow Helm chart, reported here:
https://github.com/apache/airflow/discussions/26490
Should be resolved in version 2.4.1 which should be available in the next couple of days.
I'm using Azure's Runs API to get a pipeline run result as described here:
https://learn.microsoft.com/en-us/rest/api/azure/devops/pipelines/runs/get?view=azure-devops-rest-6.0#runresult
I can see in the documentation how to get the state and final result so I can know if the run was a success or a failure. However, in case of a failure, I don't see how I can get the error that occurred in that run as a string.
How can I get the actual error which caused the pipeline run to fail?
You can use the REST API "Timeline - Get" to list the issues (error and warning) associated with a run.
Note:
This API can only list the first 10 issues. If the run has more than 10 issues, the rest will not be listed in the response. To get the complete issues, you can use the API "Builds - Get Build Log" or "Logs - Get" to get the complete logs that contains the complete issues.
[UPDATE]
The buildId is same as the runId, and you can find it from the URL of the pipeline (build) run.
The timelineId is not required in the API request, you can use the request URI like as below.
GET https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId}/timeline/?api-version=6.0
I have a node server running on Google Cloud Run. Now I want to enable stackdriver tracing. When I run the service locally, I am able to get the traces in the GCP. However, when I run the service as Google Cloud Run, I am getting an an error:
"#google-cloud/trace-agent ERROR TraceWriter#publish: Received error with status code 403 while publishing traces to cloudtrace.googleapis.com: Error: The request is missing a valid API key."
I made sure that the service account has tracing agent role.
First line in my app.js
require('#google-cloud/trace-agent').start();
running locally I am using .env file containing
GOOGLE_APPLICATION_CREDENTIALS=<path to credentials.json>
According to https://github.com/googleapis/cloud-trace-nodejs These values are auto-detected if the application is running on Google Cloud Platform so, I don't have this credentials on the gcp image
There are two challenges to using this library with Cloud Run:
Despite the note about auto-detection, Cloud Run is an exception. It is not yet autodetected. This can be addressed for now with some explicit configuration.
Because Cloud Run services only have resources until they respond to a request, queued up trace data may not be sent before CPU resources are withdrawn. This can be addressed for now by configuring the trace agent to flush ASAP
const tracer = require('#google-cloud/trace-agent').start({
serviceContext: {
service: process.env.K_SERVICE || "unknown-service",
version: process.env.K_REVISION || "unknown-revision"
},
flushDelaySeconds: 1,
});
On a quick review I couldn't see how to trigger the trace flush, but the shorter timeout should help avoid some delays in seeing the trace data appear in Stackdriver.
EDIT: While nice in theory, in practice there's still significant race conditions with CPU withdrawal. Filed https://github.com/googleapis/cloud-trace-nodejs/issues/1161 to see if we can find a more consistent solution.
I am desperately trying to debug an error 500 only when I try to update an object from my xamarin.Forms offline DB to Azure. I am using Azure Mobile Client.
I set all the logging to ON in azure, then I downloaded the log. I can see the generic error, but nothing useful.
<failedRequest url="https://MASKED:80/tables/Appel/9A3342A2-0598-4126-B0F6-2999B524B4AE"
siteId="Masked"
appPoolId="Masked"
processId="6096"
verb="PATCH"
remoteUserName=""
userName=""
tokenUserName="IIS APPPOOL\Masked"
authenticationType="anonymous"
activityId="{80000063-0000-EA00-B63F-84710C7967BB}"
failureReason="STATUS_CODE"
statusCode="500"
triggerStatusCode="500"
timeTaken="625"
xmlns:freb="http://schemas.microsoft.com/win/2006/06/iis/freb"
>
The table that failed is the only one I extend with some virtual runtime calculated field of navigation field. But I add the [JsonIgnore] to stop AzureService to create field in the local DB (that work) or send it on the wire to the server. But I always got the 500 error, not exception when debugging the c# Azure backend too.
How I can find the stack trace or the "deep" reason for this 500 error in my backend?
For C# Mobile App backend, you could add the following code in the ConfigureMobileApp method of your Startup.MobileApp.cs file for including error details and return to your client side.
config.IncludeErrorDetailPolicy = IncludeErrorDetailPolicy.Always;
You could just capture the exception in your mobile application or leverage fiddler to capture the network traces when invoking the PATCH operation to retrieve the detailed error message.
Moreover, you are viewing the Failed Request Traces log, you need to check the Application logs. Details you could follow Enable diagnostics logging for web apps in Azure App Service.
I am just trying out Application Insights availability tests.
I let them run over night to our public website and got only "fails"
When I drill in to the separate test runs, I see that the server responded with "200 OK", sending the expected html markup.
But in "Exceptins" it says
System.UriFormatException: Invalid URI: The format of the URI could not be determined.System.UriFormatException: Invalid URI: The format of the URI could not be determined.
at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)
at Microsoft.VisualStudio.TestTools.WebTesting.HtmlDocument.ParseForDependentRequests(Boolean urlsOnly)
at Microsoft.VisualStudio.TestTools.WebStress.WebTestCaseVariation.BuildDependentRequestsList(WebTestInstrumentedTransaction instrumentedTransaction)
at Microsoft.VisualStudio.TestTools.WebStress.WebTestCaseVariation.AddParsedDependentRequests(ExecutionState executionState)
Does anyone have an idea what's going on there? What am I missing?
Most likely this exception occurs when webtest engine tries to parse dependent requests information to validate resources. Either webtest incorrectly parses dependency resource or web page contains an invalid reference.
WebTest should give a better error explaining to which resource it is complaining.
To workaround this problem (if it is acceptable) - you can disable "Parse dependent requests" option (see below). Note - webtest will not be testing dependent resources.