I'm fairly new to Groovy & SoapUING. I hope someone can help me figure out and fix this error. Thanks!
What I'm trying to do: Iterate through each db row item in a table and use that as input to make a HTTPBuilder request (GET or POST) either as a query in path (add baseURL/path/hello) or through parameters(add baseURL/path?searchNode="hello"). The baseURL is something like this https://search-test-env.ser.com.
Where I'm getting stuck: When I try to post a request through HTTPBuilder.
Error: PKIX Path Building Failed
Other related information:
Using ReadyAPI to run the scripts. Code is in Groovy.
Recently imported the httpbuilder jar into ReadyAPI/lib folder along with some dependencies. Of the dependencies available with httpbuilder, ReadyAPI already had few so I only picked up the ones missing. Additional jar names: ojdbc6, ojdbc6_g, signpost-commonshttp4-1.2.1.1, ezmorph-1.0.6, json-lib-2.3-jdk15, xml-resolver-1.2, signpost-core-1.2.1.1, appengine-api-1.0-sdk-1.3.8, nekohtml-1.9.16, http-builder-0.7, http-builder-0.7-sources, http-builder-0.7-javadoc.
The Service works with a manual request without groovy (simple GET on the baseURL/path) and it also works with query by string or by parameter.
The certificate is already available in the Keystore. Tried using keytool (available in ReadyAPI/bin folder) command through cmd but receiving a filenotfound error). Tried to import into the ReadyAPI\jre\lib\security.
Code:
def qryPrmRqstHTTPBldr( pBaseUrl, pPath, pQuery, pMethod ) {
def ret = null
def http = new HTTPBuilder()
def meth
if ( pMethod == 'GET') {
meth = Method.GET
}
else if ( pMethod == 'POST') {
meth = Method.POST
}
// perform a GET/POST request, expecting TEXT response
http.request(pBaseUrl, meth, ContentType.TEXT) { req ->
uri.path = pPath
uri.query = pQuery
/*
headers.'User-Agent' = 'Apache-HttpClient/4.5.1 (Java/1.8.0_66)'
headers.'Host' = 'xxx-xx-xxx.xxx.xxx'
headers.'Accept-Encoding' = 'gzip,deflate'
headers.'Connection' = 'Keep-Alive'
*/
log.info System.getProperty("java.runtime.version")
log.info System.getProperty("javax.net.ssl.trustStore")
log.info System.getProperty("javax.net.ssl.keyStore")
log.info System.getProperty("java.home")
log.info System.getProperty("java.class.path")
// response handler for a success response code
response.success = { resp, reader ->
println "response status: ${resp.statusLine}"
println 'Headers: -----------'
ret = reader.getText()
println 'Response data: -----'
println ret
println '--------------------'
}
}
return ret
}
Running this code throws the PKIX Path Building Failed error (no stack trace available) and the content for each property:
System.getProperty("java.runtime.version") // 1.8.0_66-b17
System.getProperty("javax.net.ssl.trustStore") // null
System.getProperty("javax.net.ssl.keyStore") // null
System.getProperty("java.home") // c:\program files\smartbear\readyapi-1.6.0\jre
System.getProperty("java.class.path") // C:\Program Files\SmartBear\ReadyAPI-1.6.0\.install4j\i4jruntime.jar;......
The problem
PKIX path building failed error is thrown when your client can not construct a trust path to your server certificate. This basically means that in your trust store there isn't the certificate authority which issues the end entity certificate or a intermediate certificate for your server certificate; and it's needed to build the certificate chain.
In your case javax.net.ssl.trustStore it's not defined, due System.getProperty("javax.net.ssl.trustStore") is null; so from JSSE
If the system property: javax.net.ssl.trustStore is defined, then the
TrustManagerFactory attempts to find a file using the filename
specified by that system property, and uses that file for the
KeyStore. If the javax.net.ssl.trustStorePassword system property is
also defined, its value is used to check the integrity of the data in
the truststore before opening it. If javax.net.ssl.trustStore is
defined but the specified file does not exist, then a default
TrustManager using an empty keystore is created.
If the javax.net.ssl.trustStore system property was not specified,
then if the file <java-home>/lib/security/jssecacerts exists, that
file is used. (See The Installation Directory for
information about what refers to.)
Otherwise, If the file
<java-home>/lib/security/cacerts exists, that file is used.
The solution
Then to solve the problem you need to add the required certificates to JRE_HOME/lib/security/jssecacerts if exists or to JRE_HOME/lib/security/cacerts if not from the Java which are you using to run the Groovy scripts.
The correct way: Add the certificate authority of your server certificate in the trust store. Then make sure that your server correctly serves all certificate chain until CA: intermediate certificates (if needed) until the end entity server certificate.
The brute force way: If you've some problems setting up the trust store or you're not sure if the server is pushing the chain certificates... simply put directly the server certificate in the trust store. This is not the "purist" way but since you're only testing this will works.
How to do it
Download the CA server certificate and load it to the trust store using keytool with the follow command (the default password for cacerts is changeit):
keytool -import -alias <someAlias> -file <certificatePath> -keystore <trustStorePath>
Here there is one of my answers with a bit more detail about how to load the certificate.
Related
I'm writing backend code to verify a JWS from Google's SafetyNet API, in Node.JS.
I was surprised to not find a readily available module for this, so I started looking at some simple verification of the JWS using available libraries:
First of all, Google says these are the steps required:
Extract the SSL certificate chain from the JWS message.
Validate the SSL certificate chain and use SSL hostname matching to verify that the leaf certificate was issued to the hostname
attest.android.com.
Use the certificate to verify the signature of the JWS message.
Check the data of the JWS message to make sure it matches the data within your original request. In particular, make sure that the
timestamp has been validated and that the nonce, package name, and
hashes of the app's signing certificate(s) match the expected values.
(from https://developer.android.com/training/safetynet/attestation#verify-attestation-response)
I found node-jose which offered a simple interface to verify JWS's, and it has an option to allow an embedded key.
I'm trying to understand exactly what this process does and if it's sufficient for verifying the authenticity of the JWS?
const {JWS} = require('node-jose');
const result = await JWS.createVerify({allowEmbeddedKey: true}).verify(jws);
if (result.key.kid === 'attest.android.com') {
// Are we good to go or do we manually need to verify the certificate chain further?
}
Does using the embedded key indeed validate the embedded certificate chain x5c using the root CA, and the signature against the certificate? Or do I need to explicitly obtain a public key from Google to verify the certificate separately?
Then, a somewhat related question concerns Google's API for performing this validation: there is an API https://www.googleapis.com/androidcheck/v1/attestations/verify?key=... that performs this exact operation, but it seems to have been removed from Google's docs, and can only be found referenced in dated articles and SO answers about SafetyNet such as this one which seems to suggest that this API is only for testing, and in production you should perform the certificate verification yourself. Does anyone know if this API is good for production use or not? If everyone is meant to manually verify the JWS, I find it slightly surprising that Google wouldn't offer more documentation and code examples since this process is quite error-prone, and mistakes could have serious effects? So far I've only found some 3rd party examples in Java, but no server-side code examples from Google.
Here are the steps that you would need to perform as recommended by Google.
Definitely feel free to go through all the reference links to understand the process a bit better. Do look into each library functions used here to know what they are doing and if that is exactly what you want them to do. I've written pseudocode to explain the steps. You might have to run them on a sample attestation token to test them out and change a few things accordingly.
It would also be good to look at the whole node implementation of SafetyNet in one place.
// following steps should be performed
// 1. decode the JWS
// 2. the source of the first certificate in x5c array of jws header
// should be attest.google.com
// 3. to make sure if the JWS was not tampered with, validate the signature of JWS (how signature verification is done is explained in the reference links)
// with the certificate whose source we validated
// 4. if the signature was valid, we need to know if the certificate was valid by
// explicitly checking the certificate chain
// 5. Validate the payload by matching the package name, apkCertificateDigest
// and nonce value (apkCertificateDigest is base64 encoding of the hash of signing app's certificate)
// 6. and now you can trust the ctsProfileMatch and BasicIntegrity flags
// let's see some code in node, though this will not run as-is,
// it provides an outline on how to do it and which functions to consider when implementing
const pki = require('node-forge').pki;
const jws = require('jws');
const pem = require("pem");
const forge = require('node-forge');
const signedAttestation = "Your signed attestation here";
function deviceAttestationCheck(signedAttestation) {
// 1. decode the jws
const decodedJws = jws.decode(signedAttestation);
const payload = JSON.parse(decodedJws.payload);
// convert the certificate received in the x5c array into valid certificates by adding
// '-----BEGIN CERTIFICATE-----\n' and '-----END CERTIFICATE-----'
// at the start and end respectively for each certificate in the array
// and by adding '\n' at every 64 char
// you'll have to write your own function to do the simple string reformatting
// get the x5c certificate array
const x5cArray = decodedJws.header.x5c;
updatedX5cArray = doTheReformatting(x5cArray);
// 2. verify the source to be attest.google.com
certToVerify = updatedX5cArray[0];
const details = pem.readCertificateInfo(certToVerify);
// check if details.commanName === "attest.google.com"
const certs = updatedX5cArray.map((cert) => pki.certificateFromPem(cert));
// 3. Verify the signature with the certificate that we received
// the first element of the certificate(certs array) is the one that was issued to us, so we should use that to verify the signature
const isSignatureValid = jws.verify(signedAttestation, 'RS256', certs[0]);
// 4. to be sure if the certificate we used to verify the signature is the valid one, we should validate the certificate chain
const gsr2Reformatted = doTheReformatting(gsr2);
const rootCert = pki.certificateFromPem(gsr2Reformatted);
const caStore = pki.createCaStore([rootCert]);
// NOTE: this pki implementation does not check for certificate revocation list, which is something that you'll need to do separately
const isChainValid = pki.verifyCertificateChain(caStore, certs);
// 5. now we can validate the payload
// check the timestamps, to be within certain time say 1 hour
// check nonce value, to contain the data that you expect, refer links below
// check apkPackageName to be your app's package name
// check apkCertificateDigestSha256 to be from your app - quick tip -look at the function below on how to generate this
// finally you can trust the ctsProfileMatch - true/false depending on strict security need and basicIntegrity - true, minimum to check
}
// this function takes your signing certificate(should be of the form '----BEGIN CERT....data...---END CERT...') and converts into the SHA256 digest in hex, which looks like - 92:8H:N9:84:YT:94:8N.....
// we need to convert this hex digest to base64
// 1. 92:8H:N9:84:YT:94:8N.....
// 2. 928hn984yt948n - remove the colon and toLowerCase
// 3. encode it in base64
function certificateToSha256DigestHex(certPem) {
const cert = pki.certificateFromPem(certPem);
const der = forge.asn1.toDer(pki.certificateToAsn1(cert)).getBytes();
const m = forge.md.sha256.create();
m.start();
m.update(der);
const fingerprint = m.digest()
.toHex()
.match(/.{2}/g)
.join(':')
.toUpperCase();
return fingerprint
}
// 92:8H:N9:84:YT:94:8N => 928hn984yt948n
function stringToHex(sha256string) {
return sha256string.split(":").join('').toLowerCase();
}
// this is what google sends you in apkCertificateDigestSha256 array
// 928hn984yt948n => "OIHf9wjfjkjf9fj0a="
function hexToBase64(hexString) {
return Buffer.from(hexString, 'hex').toString('base64')
}
All the articles that helped me:
Summary for the steps - Here
explanation in depth with implementation - Here
Things you should keep in mind - Here
checklist from google to do it correctly - Here
Deep Dive into the process - Here
This is my code:
public class DatabaseHandler : MonoBehaviour
{
string url = "https://fakeid.firebaseio.com/";
void Start()
{
StartCoroutine(GetLevelsCoroutine());
}
IEnumerator GetLevelsCoroutine()
{
using (UnityWebRequest www = UnityWebRequest.Get(url))
{
www.SetRequestHeader("X-Firebase-Decoding", "1");
yield return www.SendWebRequest();
if (www.isDone)
{
Debug.Log(www.error);
string result = www.downloadHandler.text;
Debug.Log(result);
}
}
}
}
The result variable is null and the www.error is "unknown error"
I have been trying different things in order to fix this but, I just can't figure out what's causing this error, since it's just a generic error.
I have also read that this may be an unitywebrequest bug, if it is so, are there any alternatives?
Finally found the solution to the problem. (It only occurs on some Linux OS)
Unity only officially supports Ubuntu Linux, so it is looking (and failing to find) the certificate store where it would expect it to be. You can work around on Fedora by creating a symbolic link:
mkdir -p /etc/ssl/certs && ln -s /etc/pki/tls/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt
This is the source where I got it from: https://forum.unity.com/threads/ubuntu-headless-build-tls-handshake-fails.546704/
It seems like you're trying to get JSON from the Firebase Realtime Database through its REST API. Requests to the REST API must end in .json, otherwise Firebase interprets them as requests to open the console on that location. So UnityWebRequest.Get(url+.json)
Suppose I'm using a third party network library that I don't want to modify. It uses the standard http.Request interface to make some HTTPS requests.
I'm running it on an embedded Linux instance that doesn't have a certificate roots installed, and the only directory I can access is /data. This means you get the error:
Get https://example.com/: x509: failed to load system roots and no roots provided
Is there any way to actually provide roots? As far as I can tell Go looks in these directories for X509 certificate roots (see also):
var certFiles = []string{
"/etc/ssl/certs/ca-certificates.crt", // Debian/Ubuntu/Gentoo etc.
"/etc/pki/tls/certs/ca-bundle.crt", // Fedora/RHEL
"/etc/ssl/ca-bundle.pem", // OpenSUSE
"/etc/pki/tls/cacert.pem", // OpenELEC
}
var certDirectories = []string{
"/system/etc/security/cacerts", // Android
}
As I said, I don't have access to those, and the root pool seems to be private so you can't append to it:
var (
once sync.Once
systemRoots *CertPool
)
func systemRootsPool() *CertPool {
once.Do(initSystemRoots)
return systemRoots
}
Is this just impossible with Go?
Something like this seems to work (I actually only need one certificate chain, which you can get from Firefox easily by clicking on the SSL lock icon, More Information, View Certificate, Details, Export, Change type to "X509 certificate with chain (PEM)".
func initCerts() error {
certs := x509.NewCertPool()
pemData, err := ioutil.ReadFile("api.example.crt")
if err != nil {
return err
}
certs.AppendCertsFromPEM(pemData)
newTlsConfig := &tls.Config{}
newTlsConfig.RootCAs = certs
defaultTransport := http.DefaultTransport.(*http.Transport)
defaultTransport.TLSClientConfig = newTlsConfig
return nil
}
I'm not certain, but the docs suggest that you must do that before using any TLS functions, so I put it at the start of my main().
The Azure portal will not let me add a Cer file. I found a post saying you can type the name in, you can only do this from the open dialogue and it does not work
So I am trying to convert the file using code I have placed in my Global.asax. I have copied the CER file into my temp directory off the root of C drive
Here is the code and Error I get :
string file = #"C:\temp\SVRSecureG3.cer";
var cert = System.Security.Cryptography.X509Certificates.X509Certificate2.CreateFromCertFile(file);
// Error occurs on the line below, I get : {"Unable to cast object of type 'System.Security.Cryptography.X509Certificates.X509Certificate' to type 'System.Security.Cryptography.X509Certificates.X509Certificate2'."}
var bytes = ((System.Security.Cryptography.X509Certificates.X509Certificate2)cert).Export(System.Security.Cryptography.X509Certificates.X509ContentType.Pfx, "p");
var fs = File.Create(#"C:\temp\SVRSecureG3.pfx");
using (fs)
{
fs.Write(bytes, 0, bytes.Length);
fs.Flush();
}
I would actually just recommend converting your .cer to a .pfx. Use OpenSSL to do this, I have done this numerous times to upload certificates to Azure.
Download
http://slproweb.com/products/Win32OpenSSL.html
Usage
create a pfx file from a .cer and a .pem file
I'm having a heck of a time here trying to use google OAuth to authenticate users in my node express app. I can successfully do the OAuth, which returns a response like so:
{
access_token: 'token string',
id_token: 'id.string',
expires_in: 3599,
token_type: "Bearer"
}
This all makes sense, but I can't for the life of me figure out how to decode the JWT. I am a bit inexperienced in all this, so this is all a bit foreign to me.
Following the instructions listed here: https://developers.google.com/accounts/docs/OAuth2Login#validatinganidtoken I am attempting to decode the JWT locally in my node app.
I installed https://github.com/hokaccha/node-jwt-simple in my node environment.
And I'm pretty certain I need to use this certificate (https://www.googleapis.com/oauth2/v1/certs) in all this somehow to decode it, but I am at a bit of a loss here. I don't really understand how I get the certificate into my node app, and after that how to use it with node-jwt-simple. And I also don't really understand how I know when I need to pull a fresh certificate, vs using a cached one.
Anyone out there with some experience in this that can help me out?
Thanks for any help. I'm totally at a loss at this point.
** Update **
So I have made some progress... Kind of.
By calling jwt.decode(id_token, certificate, true); I am able to successfully decode the token. Even if the certificate var is an empty object {}. This leaves me with 3 questions still.
1: What is the best way to get the certificate into my express app using the url from google?
2: How will I know when I need to pull in a fresh version of it?
3: It seems like passing in true for noVerify (3rd arg in jwt.decode) is a terrible idea. How can I get that to work without passing that in?
It looks like perhaps jwt-simple is expecting hs256 and the token is using rs256.
Again, I'm super inexperienced in this, so I may be way off base here.
* UPDATE *
Thanks to the help from Nat, I was able to get this working!
I think I tried every single JWT and JWS node module out there. What I finally landed on is as follows:
I found that none of the modules that I looked at did quite what I wanted out of the box. I created the following jwt decoding helper methods that I am using to decode the id_token, so I can get the kid from the header.
module.exports = {
decodeJwt: function (token) {
var segments = token.split('.');
if (segments.length !== 3) {
throw new Error('Not enough or too many segments');
}
// All segment should be base64
var headerSeg = segments[0];
var payloadSeg = segments[1];
var signatureSeg = segments[2];
// base64 decode and parse JSON
var header = JSON.parse(base64urlDecode(headerSeg));
var payload = JSON.parse(base64urlDecode(payloadSeg));
return {
header: header,
payload: payload,
signature: signatureSeg
}
}
}
function base64urlDecode(str) {
return new Buffer(base64urlUnescape(str), 'base64').toString();
};
function base64urlUnescape(str) {
str += Array(5 - str.length % 4).join('=');
return str.replace(/\-/g, '+').replace(/_/g, '/');
}
I am using this decoding to determine if I need to pull in a new public cert from: https://www.googleapis.com/oauth2/v1/certs
Then I am using that public cert and node-jws (https://github.com/brianloveswords/node-jws) jws.verify(id_token, cert) to verify the signature!
Hooray!
Thanks again for the extra explanation you gave in your response. That went a long way in helping me understand what I was even trying to do. Hope this might help others too.
From the specification point of view, what you are encountering is [OpenID Connect].
id_token is a [JWS] signed [JWT]. In this case, it is a "." separated string with three components. The first portion is the header. The second is the payload. The third is the signature. Each of them are Base64url encoded string.
When you decode the header, you will get something like:
{"alg":"RS256","kid":"43ebb53b0397e7aaf3087d6844e37d55c5fb1b67"}
The "alg" indicates that the signature algorithm is RS256, which is defined in [JWA].
The "kid" indicates the key id of the public key that corresponds to the key used to sign.
Now I am ready to answer some of your questions:
2: How will I know when I need to pull in a fresh version of it?
When the kid of the cached cert file (a [JWK] file) does not match the kid in the header, fetch a new cert file. (BTW, the URL from which you pull the certs are called x5u.)
3: It seems like passing in true for noVerify (3rd arg in jwt.decode)
is a terrible idea. How can I get that to work without passing that
in?
Indeed. Perhaps you might want to look at another library such as kjur.github.io/jsjws/ .
References
[OpenID Connect] openid.bitbucket.org/openid-connect-core-1_0.html
[JWS] tools.ietf.org/html/draft-ietf-jose-json-web-signature
[JWT] tools.ietf.org/html/draft-ietf-oauth-json-web-token‎
[JWK] tools.ietf.org/html/draft-ietf-oauth-json-web-keys
[JWA] tools.ietf.org/html/draft-ietf-jose-json-web-algorithms