I have an artillery config setup to test two different versions of my API gateway (Apollo federated Graphql gateway). The V2 gateway URL returns 404 with artillery, but from the browser and Postman, it is accessible. I am quite confused as to why this is the case. What could be the issue here? The V1 URL of the gateway is accessible by artillery, but when it hits the V2 URL, it returns a 404. Here is my config file
config:
environments:
v2:
target: "https://v2.com/graphql" # not accessible by artillery but works with postman and browser
phases:
- duration: 60
arrivalRate: 5
name: Warm up
- duration: 120
arrivalRate: 5
rampTo: 50
name: Ramp up load
- duration: 300
arrivalRate: 50
name: Sustained load
v1:
target: "https://v1.com/graphql" # accessible by artillery and other agents
phases:
- duration: 60
arrivalRate: 5
name: Warm up
- duration: 120
arrivalRate: 5
rampTo: 50
name: Ramp up load
payload:
path: "test-user.csv"
skipHeader: true
fields:
- "email"
- "_id"
-
path: "test-campaign.csv"
skipHeader: true
fields:
- "campaignId"
scenarios:
- name: "Test"
flow:
- post:
url: "/"
headers:
Authorization: Bearer <token>
json:
query: |
query Query {
randomQuery {
... on Error {
message
statusCode
}
... on Response {
res {
__typename
}
}
Any help would be appreciated. Thank you.
Related
I have an AKS cluster with a Node.js server connecting to a Neo4j-standalone instance all deployed with Helm.
I installed an ingress-nginx controller, referenced a default Let's Encrypt certificate and habilitated TPC ports with Terraform as
resource "helm_release" "nginx" {
name = "ingress-nginx"
repository = "ingress-nginx"
# repository = "https://kubernetes.github.io/ingress-nginx"
chart = "ingress-nginx/ingress-nginx"
namespace = "default"
set {
name = "tcp.7687"
value = "default/cluster:7687"
}
set {
name = "tcp.7474"
value = "default/cluster:7474"
}
set {
name = "tcp.7473"
value = "default/cluster:7473"
}
set {
name = "tcp.6362"
value = "default/cluster-admin:6362"
}
set {
name = "tcp.7687"
value = "default/cluster-admin:7687"
}
set {
name = "tcp.7474"
value = "default/cluster-admin:7474"
}
set {
name = "tcp.7473"
value = "default/cluster-admin:7473"
}
set {
name = "controller.extraArgs.default-ssl-certificate"
value = "default/tls-secret"
}
set {
name = "controller.service.externalTrafficPolicy"
value = "Local"
}
set {
name = "controller.service.annotations.service.beta.kubernetes.io/azure-load-balancer-internal"
value = "true"
}
set {
name = "controller.service.loadBalancerIP"
value = var.public_ip_address
}
set {
name = "controller.service.annotations.service.beta.kubernetes.io/azure-dns-label-name"
value = "xxx.westeurope.cloudapp.azure.com"
}
set {
name = "controller.service.annotations.service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path"
value = "/healthz"
}
}
I then have an Ingress with paths pointing to Neo4j services so on https://xxx.westeurope.cloudapp.azure.com/neo4j-tcp-http/browser/ I can get to the browser.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-service
namespace: default
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$2$3$4
# nginx.ingress.kubernetes.io/rewrite-target: /
# certmanager.k8s.io/acme-challenge-type: http01
nginx.ingress.kubernetes/cluster-issuer: letsencrypt-issuer
ingress.kubernetes.io/ssl-redirect: "true"
# kubernetes.io/tls-acme: "true"
spec:
ingressClassName: nginx
tls:
- hosts:
- xxxx.westeurope.cloudapp.azure.com
secretName: tls-secret
rules:
# - host: xxx.westeurope.cloud.app.azure.com #dns from Azure PublicIP
### Node.js server
- http:
paths:
- path: /(/|$)(.*)
pathType: Prefix
backend:
service:
name: server-clusterip-service
port:
number: 80
- http:
paths:
- path: /server(/|$)(.*)
pathType: Prefix
backend:
service:
name: server-clusterip-service
port:
number: 80
##### Neo4j
- http:
paths:
# 502 bad gateway
# /any character 502 bad gatway
- path: /neo4j-tcp-bolt(/|$)(.*)
pathType: Prefix
backend:
service:
# neo4j chart
# name: cluster
# neo4j-standalone chart
name: neo4j
port:
# name: tcp-bolt
number: 7687
- http:
paths:
# /browser/ show browser
#/any character shows login to xxx.westeurope.cloudapp.azure.com:443 from https, :80 from http
- path: /neo4j-tcp-http(/|$)(.*)
pathType: Prefix
backend:
service:
# neo4j chart
# name: cluster
# neo4j-standalone chart
name: neo4j
port:
# name: tcp-http
number: 7474
- http:
paths:
- path: /neo4j-tcp-https(/|$)(.*)
# 502 bad gateway
# /any character 502 bad gatway
pathType: Prefix
backend:
service:
# neo4j chart
# name: cluster
# neo4j-standalone chart
name: neo4j
port:
# name: tcp-https
number: 7473
I can get to the Neo4j Browser on the https://xxx.westeurope.cloudapp.azure.com/neo4j-tcp-http/browser/ but using the Connect Url bolt+s//server.bolt it won't connect to the server with the error ServiceUnavailable: WebSocket connection failure. Due to security constraints in your web browser, the reason for the failure is not available to this Neo4j Driver..
Now I'm guessing that is because Neo4j bolt connector is not using the Certificate used by the ingress-nginxcontroller.
vincenzocalia#vincenzos-MacBook-Air helm_charts % kubectl describe secret tls-secret
Name: tls-secret
Namespace: default
Labels: controller.cert-manager.io/fao=true
Annotations: cert-manager.io/alt-names: xxx.westeurope.cloudapp.azure.com
cert-manager.io/certificate-name: tls-certificate
cert-manager.io/common-name: xxx.westeurope.cloudapp.azure.com
cert-manager.io/ip-sans:
cert-manager.io/issuer-group:
cert-manager.io/issuer-kind: ClusterIssuer
cert-manager.io/issuer-name: letsencrypt-issuer
cert-manager.io/uri-sans:
Type: kubernetes.io/tls
Data
====
tls.crt: 5648 bytes
tls.key: 1679 bytes
I tried to use it overriding the chart values, but then the Neo4j driver from Node.js server won't connect to the server ..
ssl:
# setting per "connector" matching neo4j config
bolt:
privateKey:
secretName: tls-secret # we set up the template to grab `private.key` from this secret
subPath: tls.key # we specify the privateKey value name to get from the secret
publicCertificate:
secretName: tls-secret # we set up the template to grab `public.crt` from this secret
subPath: tls.crt # we specify the publicCertificate value name to get from the secret
trustedCerts:
sources: [ ] # a sources array for a projected volume - this allows someone to (relatively) easily mount multiple public certs from multiple secrets for example.
revokedCerts:
sources: [ ] # a sources array for a projected volume
https:
privateKey:
secretName: tls-secret
subPath: tls.key
publicCertificate:
secretName: tls-secret
subPath: tls.crt
trustedCerts:
sources: [ ]
revokedCerts:
sources: [ ]
Is there a way to use it or should I setup another certificate just for Neo4j? If so what would it be the dnsNames to set on it?
Is there something else I'm doing wrong?
Thank you very much.
From what I can gather from your information, the problem seems to be that you're trying to expose the bolt port behind an ingress. Ingresses are implemented as an L7 (protocol aware) reverse proxy and manage load-balancing etc. The bolt protocol has its load balancing and routing for cluster applications. So you will need to expose the network service directly for every instance of neo4j you are running.
Check out this part of the documentation for more information:
https://neo4j.com/docs/operations-manual/current/kubernetes/accessing-neo4j/#access-outside-k8s
Finally after a few days of going in circles I found what the problems were..
First using a Staging certificate will cause Neo4j bolt connection to fail, as it's not Trusted, with error:
ServiceUnavailable: WebSocket connection failure. Due to security constraints in your web browser, the reason for the failure is not available to this Neo4j Driver. Please use your browsers development console to determine the root cause of the failure. Common reasons include the database being unavailable, using the wrong connection URL or temporary network problems. If you have enabled encryption, ensure your browser is configured to trust the certificate Neo4j is configured to use. WebSocket readyState is: 3
found here https://grishagin.com/neo4j/2022/03/29/neo4j-websocket-issue.html
Then I was missing to assign a general listening address to the bolt connector as it's listening by default only to 127.0.0.0:7687 https://neo4j.com/docs/operations-manual/current/configuration/connectors/
To listen for Bolt connections on all network interfaces (0.0.0.0)
so I added server.bolt.listen_address: "0.0.0.0:7687" to Neo4j chart values config.
Next, as I'm connecting the default neo4j ClusterIP service tcp ports to the ingress controller's exposed TCP connections through the Ingress as described here https://neo4j.com/labs/neo4j-helm/1.0.0/externalexposure/ as an alternative to using a LoadBalancer, the Neo4j LoadBalancer services is not needed so the services:neo4j:enabled gets set to "false", in my tests I actually found that if you leave it enabled bolt won't connect despite setting everything correctly..
Other Neo4j missing config where server.bolt.enabled : "true", server.bolt.tls_level: "REQUIRED", dbms.ssl.policy.bolt.client_auth: "NONE" and dbms.ssl.policy.bolt.enabled: "true" the complete list of config options is here https://neo4j.com/docs/operations-manual/current/reference/configuration-settings/
Neo4j chart's values for ssl config were fine.
So now I can use the (renamed for brevity) path /neo4j/browser/ to serve the Neo4j Browser app, and either the /bolt path as the browser Connect URL, or PublicIP's <DSN>:<bolt port>.
You are connected as user neo4j
to bolt+s://xxxx.westeurope.cloudapp.azure.com/bolt
Connection credentials are stored in your web browser.
Hope this explanation and the code recap below will help others.
Cheers.
ingress controller
resource "helm_release" "nginx" {
name = "ingress-nginx"
namespace = "default"
repository = "https://kubernetes.github.io/ingress-nginx"
chart = "ingress-nginx"
set {
name = "version"
value = "4.4.2"
}
### expose tcp connections for neo4j service
### bolt url connection port
set {
name = "tcp.7687"
value = "default/neo4j:7687"
}
### http browser app port
set {
name = "tcp.7474"
value = "default/neo4j:7474"
}
set {
name = "controller.extraArgs.default-ssl-certificate"
value = "default/tls-secret"
}
set {
name = "controller.service.externalTrafficPolicy"
value = "Local"
}
set {
name = "controller.service.annotations.service.beta.kubernetes.io/azure-load-balancer-internal"
value = "true"
}
set {
name = "controller.service.loadBalancerIP"
value = var.public_ip_address
}
set {
name = "controller.service.annotations.service.beta.kubernetes.io/azure-dns-label-name"
value = "xxx.westeurope.cloudapp.azure.com"
}
set {
name = "controller.service.annotations.service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path"
value = "/healthz"
}
}
Ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-service
namespace: default
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$2$3$4
ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes/cluster-issuer: letsencrypt-issuer
spec:
ingressClassName: nginx
tls:
- hosts:
- xxx.westeurope.cloudapp.azure.com
secretName: tls-secret
rules:
### Node.js server
- http:
paths:
- path: /(/|$)(.*)
pathType: Prefix
backend:
service:
name: server-clusterip-service
port:
number: 80
- http:
paths:
- path: /server(/|$)(.*)
pathType: Prefix
backend:
service:
name: server-clusterip-service
port:
number: 80
##### Neo4j
- http:
paths:
- path: /bolt(/|$)(.*)
pathType: Prefix
backend:
service:
name: neo4j
port:
# name: tcp-bolt
number: 7687
- http:
paths:
- path: /neo4j(/|$)(.*)
pathType: Prefix
backend:
service:
name: neo4j
port:
# name: tcp-http
number: 7474
Values.yaml (Umbrella chart)
neo4j-db: #chart dependency alias
nameOverride: "neo4j"
fullnameOverride: 'neo4j'
neo4j:
# Name of your cluster
name: "xxxx" # this will be the label: app: value for the service selector
password: "xxxxx"
##
passwordFromSecret: ""
passwordFromSecretLookup: false
edition: "community"
acceptLicenseAgreement: "yes"
offlineMaintenanceModeEnabled: false
resources:
cpu: "1000m"
memory: "2Gi"
volumes:
data:
mode: 'volumeClaimTemplate'
volumeClaimTemplate:
accessModes:
- ReadWriteOnce
storageClassName: neo4j-sc-data
resources:
requests:
storage: 4Gi
backups:
mode: 'share' # share an existing volume (e.g. the data volume)
share:
name: 'logs'
logs:
mode: 'volumeClaimTemplate'
volumeClaimTemplate:
accessModes:
- ReadWriteOnce
storageClassName: neo4j-sc-logs
resources:
requests:
storage: 4Gi
services:
# A LoadBalancer Service for external Neo4j driver applications and Neo4j Browser, this will create "cluster-neo4j" svc
neo4j:
enabled: false
config:
server.bolt.enabled : "true"
server.bolt.tls_level: "REQUIRED"
server.bolt.listen_address: "0.0.0.0:7687"
dbms.ssl.policy.bolt.client_auth: "NONE"
dbms.ssl.policy.bolt.enabled: "true"
startupProbe:
failureThreshold: 1000
periodSeconds: 50
ssl:
bolt:
privateKey:
secretName: tls-secret
subPath: tls.key
publicCertificate:
secretName: tls-secret
subPath: tls.crt
trustedCerts:
sources: [ ]
revokedCerts:
sources: [ ] # a sources array for a projected volume
I created moodle and mariadb containers with Docker.
Moodle: 3.11.4
Mariadb: 10.3
I am trying following webservice to execute:
client:
wwwroot: 'http://localhost:8012',
service: 'moodle_mobile_app',
token: '8faf4879d2c654f11e404095032ae382',
strictSSL: true
call:
curl "http://localhost:8012/webservice/rest/server.php?wstoken=8faf4879d2c654f11e404095032ae382&moodlewsrestformat=json&wsfunction=core_user_get_users_by_field&moodlewsrestformat=json&id=2"
but getting follwing error:
{"exception":"invalid_parameter_exception","errorcode":"invalidparameter",
"message":"Invalid parameter value detected (Missing required key in single structure:field)",
"debuginfo":"Missing required key in single structure: field"
}
I tried it same with moodle client for node
... client.call({ wsfunction: "core_user_get_users_by_field", method: "POST", args: { id: 2 } })...
but also receiving same error.
I checked API documentation and id is valid parameter for this
webservice.
Can you please help?
Issue is resolved
client.call({
method: "POST",
wsfunction: "core_user_get_users_by_field",
args: {
field: "id",
values: ["2"]
}
}).then(function(info) {
var str = JSON.stringify(info, null, 4);
console.log(str);
});
I've implemented a custom 'REQUEST' type authorizer for an API gateway which validates a JWT token passed in the 'Authorization' header. I've tested the lambda independently and it works as expected. I've also attached the authorizer to my routes and I can test it in the AWS console - again, everything seems to work (see image):
successful invoke via console
However, when I try to invoke my endpoints with the token in the Authorization header, I always receive an UNAUTHORIZED response:
{
"errors": [
{
"category": "ClientError",
"code": "UNAUTHORIZED",
"detail": "Unauthorized",
"method": "GET",
"path": "/cases",
"requestId": "004eb254-a926-45ad-96a5-ce3527621c81",
"retryable": false
}
]
}
From what I have gathered, API gateway never invokes my Authorizer as i don't see any log events in its cloudwatch. I was able to enable cloudwatch logging of my API gateway, and the only log information I see is as follows:
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| timestamp | message |
|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1578275720543 | (dac0d4f6-1380-4049-bcee-bf776ca78e5c) Extended Request Id: F2v9WFfiIAMF-9w= |
| 1578275720543 | (dac0d4f6-1380-4049-bcee-bf776ca78e5c) Unauthorized request: dac0d4f6-1380-4049-bcee-bf776ca78e5c |
| 1578275720543 | (dac0d4f6-1380-4049-bcee-bf776ca78e5c) Extended Request Id: F2v9WFfiIAMF-9w= |
| 1578275720544 | (dac0d4f6-1380-4049-bcee-bf776ca78e5c) Gateway response type: UNAUTHORIZED with status code: 401 |
| 1578275720544 | (dac0d4f6-1380-4049-bcee-bf776ca78e5c) Gateway response body: {"errors": [{"category": "ClientError","code": "UNAUTHORIZED","detail": "Unauthorized","method": "GET","path": "/cases","requestId": "dac0d4f6-1380-4049-bcee-bf776ca78e5c","retryable": false }]} |
| 1578275720544 | (dac0d4f6-1380-4049-bcee-bf776ca78e5c) Gateway response headers: {} |
| 1578275720544 | (dac0d4f6-1380-4049-bcee-bf776ca78e5c) Gateway response type: UNAUTHORIZED with status code: 401 |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
At this point I am completely stuck and not sure how to debug this further. I'm assuming something must be configured wrong but the log information I can find doesn't given any indication of what the problem is. I've also pasted a copy of my authorizers configuration in the image below:
Authorizer Configuration
Screenshot of one endpoint configured to use the authorizer
I figured out the problem I was having:
I needed to set identitySource: method.request.header.Authorization in the authorizer field of the endpoint as well as in the CF stack.
Custom Authorizer definition in raw cloudformation:
service:
name: api-base
frameworkVersion: ">=1.2.0 <2.0.0"
plugins:
- serverless-plugin-optimize
- serverless-offline
- serverless-pseudo-parameters
- serverless-domain-manager
custom:
stage: ${self:provider.stage, 'dev'}
serverless-offline:
port: ${env:OFFLINE_PORT, '4000'}
false: false
cognitoStack: marley-auth
customDomain:
domainName: ${env:BE_HOST, ''}
enabled: ${env:EN_CUSTOM_DOMAIN, self:custom.false}
stage: ${self:provider.stage, 'dev'}
createRoute53Record: true
provider:
name: aws
runtime: nodejs10.x
versionFunctions: true
apiName: public
logs:
restApi: true
stackTags:
COMMIT_SHA: ${env:COMMIT_SHA, 'NO-SHA'}
environment:
USER_POOL_ID: ${cf:${self:custom.cognitoStack}-${self:custom.stage}.UserPoolId}
CLIENT_ID: ${cf:${self:custom.cognitoStack}-${self:custom.stage}.UserPoolClientId}
timeout: 30
iamRoleStatements:
- Effect: "Allow"
Action:
- "lambda:InvokeFunction"
Resource: "*"
functions:
authorizer:
handler: handler/authorize.handler
resources:
- Outputs:
ApiGatewayRestApiId:
Value:
Ref: ApiGatewayRestApi
Export:
Name: ${self:custom.stage}-${self:provider.apiName}-ApiGatewayRestApiId
ApiGatewayRestApiRootResourceId:
Value:
Fn::GetAtt:
- ApiGatewayRestApi
- RootResourceId
Export:
Name: ${self:custom.stage}-${self:provider.apiName}-ApiGatewayRestApiRootResourceId
SharedAuthorizerId:
Value:
Ref: SharedAuthorizer
Export:
Name: ${self:custom.stage}-${self:provider.apiName}-ApiGatewaySharedAuthorizerId
- Resources:
SharedAuthorizer:
Type: AWS::ApiGateway::Authorizer
Properties:
Name: public
AuthorizerUri: !Join
- ''
- - 'arn:aws:apigateway:'
- !Ref 'AWS::Region'
- ':lambda:path/2015-03-31/functions/'
- !GetAtt
- AuthorizerLambdaFunction
- Arn
- /invocations
RestApiId: !Ref 'ApiGatewayRestApi'
Type: REQUEST
IdentitySource: method.request.header.Authorization
AuthorizerResultTtlInSeconds: '300'
DependsOn: AuthorizerLambdaFunction
ApiAuthLambdaPermission:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
FunctionName: !Ref AuthorizerLambdaFunction
Principal: apigateway.amazonaws.com
SourceArn: !Sub "arn:aws:execute-api:#{AWS::Region}:#{AWS::AccountId}:#{ApiGatewayRestApi}/authorizers/*"
DependsOn: ApiGatewayRestApi
Using the authorizer in another stack - note that I have specified IdentitySource here as well as in the definition of the authorizer - for some reason I had to do it in both places.
authorizer:
type: CUSTOM
authorizerId: ${cf:api-base-${self:custom.stage}.SharedAuthorizerId}
identitySource: method.request.header.Authorization
I have an issue with the embedded API management within a Cloud Foundry application ( node.js ) on bluemix .There is a certain path in the yaml which is not working via the gateway, please see below the relevant path from the yaml:
/socket.io/:
get:
produces:
- text/plain; charset=utf-8
parameters: []
responses:
default:
description: Definition generated from Swagger Inspector
I get 404 , not found.
The url works fine when I dont go via the gateway.
The url is https://[masked api mgd hostname]/socket.io/?EIO=3&transport=polling&t=MC0pE73
Please help.
Find attached the complete yaml below
swagger: "2.0"
info:
description: defaultDescription
version: "0.1"
title: defaultTitle
host: masked.actualEndpoint
schemes:
- https
basePath: "/"
paths:
/socket.io/:
get:
parameters:
- name: t
in: query
required: false
type: string
x-example: MC0pE73
- name: EIO
in: query
required: false
type: string
x-example: "3"
- name: transport
in: query
required: false
type: string
x-example: polling
responses:
default:
description: Definition generated from Swagger Inspector
definitions: {}
am accessing the url using https://[ masked api mangd hostname ]/socket.io/?EIO=3&transport=polling&t=MCvtHJT
I believe its the / at the end of the path ( /socket.io/ ) which is causing the gateway to fail. Any comments.
Thanks for the help in advance.
Currently using cloudformation templates to deploy a simple API to AWS as part of a POC for moving from Azure to AWS API management.
I have got everything working except i have not been able to figure out the YAML AWS extension for setting the HTTP proxy checkbox for the HTTP request.
Sample YAML below. I know this will not set that checkbox (as i have tested it and it worked minus that problem), but on this page
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-swagger-extensions-integration.html
i cannot see a extension that sets this option? Has AWS not done this yet
AWSTemplateFormatVersion: '2010-09-09'
Resources:
PlayersAPI:
Type: AWS::ApiGateway::RestApi
Properties:
Name: RAH API
Description: A demo API for testing
Body:
swagger: '2.0'
info:
title: test api
description: test api
version: 1.0.1
contact:
name: SH
email: test#mailinator.com
paths:
"/heartbeat":
get:
description: Checks the API is working
produces:
- application/json
responses:
'200':
description: API Response information
x-amazon-apigateway-integration:
type: http
responses:
default:
statusCode: '200'
httpMethod: GET
uri: https://api.example.com
This works for me:
resources:
Resources:
ProxyResource:
Type: AWS::ApiGateway::Resource
Properties:
ParentId:
Fn::GetAtt:
- ApiGatewayRestApi # our default Rest API logical ID
- RootResourceId
PathPart: "{proxy+}" # the endpoint in your API that is set as proxy
RestApiId:
Ref: ApiGatewayRestApi
ProxyMethod:
Type: AWS::ApiGateway::Method
Properties:
ResourceId:
Ref: ProxyResource
RestApiId:
Ref: ApiGatewayRestApi
HttpMethod: GET # the method of your proxy. Is it GET or POST or ... ?
MethodResponses:
- StatusCode: 200
Integration:
IntegrationHttpMethod: GET
Type: HTTP_PROXY
Uri: http://bucket.mybucket.co.s3.eu-west-1.amazonaws.com/{proxy} # the URL you want to set a proxy to
IntegrationResponses:
- StatusCode: 200
AuthorizationType: NONE