Been troubleshooting for a while and can't come up with an answer. I run get commands with no problem so list basecamps, todos, accounts, etc. All that's working fine. It's when I POST to todolists.json where I'm having problems.
I'm sending to this URL:
https://3.basecampapi.com/3266090/buckets/1940710/todosets/296431359/todolists.json
Headers:
[0] => Content-Type: application/json; charset=utf-8
[1] => User-Agent: Template Deployer (s****d#***.org)
[2] => Authorization: Bearer [working token]
[3] => If-None-Match:
POST payload:
name=foobar
And here's the response I'm getting:
[result] => Array
(
[status] => 400
[error] => Bad Request
)
[headers] => Array
(
[Server] => nginx
[Date] => Fri, 18 Nov 2016 21:05:44 GMT
[Content-Type] => application/json; charset=utf-8
[Content-Length] => 36
[Connection] => keep-alive
[X-Request-Id] => ab71cfaa-e24c-4a38-b1af-44fe8ae8d147
[X-Runtime] => 0.456205
[Strict-Transport-Security] => max-age=15552000; includeSubDomains; preload
[X-Queue-Time] => 0.0010805130004882812
[Set-Cookie] => force-primary-dc=true;Max-Age=5;Path=/;Version=1;
)
[code] => 400
[content_type] => application/json; charset=utf-8
I've tried several combinations of name, description and project (from another thread I found about someone else's problem), to no avail. I don't even know what else to try.
Any Ideas?
And it's because I'm not a smart person :)
Answer: the payload needs to be json encoded: {"content":"Do this!"} rather than url encoded: name=foobar
Problem Solved!
Related
On the command line, I can do a request like: curl -I -H "Fastly-Debug: 1"
and it will return a lot of helpful information from the CDN serving that URL, in this case, Fastly:
cache-control: public, max-age=0, must-revalidate
last-modified: Tue, 20 Apr 2021 21:17:46 GMT
etag: "4c5cb3eb0ddb001584dad329b8727a9a"
content-type: text/html
server: AmazonS3
surrogate-key: /v3.6/tutorial/nav/alerts-and-monitoring/
accept-ranges: bytes
date: Fri, 30 Apr 2021 20:50:15 GMT
via: 1.1 varnish
age: 0
fastly-debug-path: (D cache-lga21923-LGA 1619815815) (F cache-lga21940-LGA 1619815815)
fastly-debug-ttl: (M cache-lga21923-LGA - - 0)
fastly-debug-digest: 04c3e527819b6a877de6577f7461e132b97665100a63ca8f667d87d049092233
x-served-by: cache-lga21923-LGA
x-cache: MISS
x-cache-hits: 0
x-timer: S1619815815.944515,VS0,VE136
vary: Accept-Encoding
content-length: 65489
How can I do this in node?
This is my attempt:
const headers = {
'Fastly-Key': environment.getFastlyToken(),
'Accept': 'application/json',
'Content-Type': 'application/json',
'Fastly-Debug': 1
};
async retrieveSurrogateKey(url) {
console.log("this is the url: ", url)
try {
request({
method: `GET`,
url: url,
headers: headers,
}, function(err, response, body) {
if (err){
console.trace(err)
}
console.log(request.headers)
})
} catch (error) {
console.log("error in retrieval: ", error)
}
}
Is there a way for me to pass in the -I and -H flags?
The -H (header) flag allows you to specify a custom header in cURL. Your code already does this - great! All that's left is emulating the -I (head) flag.
From cURL manpage for the -I option:
Fetch the headers only! HTTP-servers feature the command HEAD which this uses to get nothing but the header of a document.
To use the HEAD method, you need to specify it instead of GET:
method: `HEAD`
And finally, the headers returned by the server in the response can be obtained from response.headers.
Etag is supported in SDK: https://sap.github.io/cloud-sdk/docs/java/features/odata/use-typed-odata-v4-client-in-sap-cloud-sdk-for-java/#handling-of-etags
So experimenting it by using BusinessPartner entity in S/4 EX.
But seems there's no If-Match header:
How come the header doesn't show up - any prerequisite with etag?
(entering on behalf of the implementation partner team)
I checked the VersionIdentifier of the response and it was not set to a value.
I also checked the response's JSON __metadeta and header, but there were no values that appeared to correspond to the ETag value.
[Code]
BusinessPartner bp1 = new DefaultBusinessPartnerService().getBusinessPartnerByKey(bpId).execute(dest);
log.debug("get 1: {}", bp1);
log.debug("get 1 VersionIdentifier: {}", bp1.getVersionIdentifier());
bp1.setOrganizationBPName1("SCP Update 1st:" + System.currentTimeMillis());
ODataUpdateResult result1 = new DefaultBusinessPartnerService().updateBusinessPartner(bp1).execute(dest);
log.debug("Update1 Http Status: {}", result1.getHttpStatusCode());
bp1.setOrganizationBPName1("SCP Update 2nd:" + System.currentTimeMillis());
bp1.setVersionIdentifier("dummy");
ODataUpdateResult result2 = new DefaultBusinessPartnerService().updateBusinessPartner(bp1).execute(dest);
log.debug("Update2 Http Status: {}", result2.getHttpStatusCode());
[Log]
get 1: BusinessPartner(super=VdmObject(customFields={}, changedOriginal...
get 1 VersionIdentifier: None
Update1 Http Status: 204
Update2 Http Status: 204
[GET Response JSON(__metadata) / Response Header]
(It has masked the IP address.)
"__metadata": {
"id": "https://xxx.xxx.xxx.xxx:xxxxxx/sap/opu/odata/sap/API_BUSINESS_PARTNER/A_BusinessPartner('1000001')",
"uri": "https://xxx.xxx.xxx.xxx:xxxxxx/sap/opu/odata/sap/API_BUSINESS_PARTNER/A_BusinessPartner('1000001')",
"type": "API_BUSINESS_PARTNER.A_BusinessPartnerType"
},
HTTP/1.1 200 OK
content-type: application/json; charset=utf-8
content-length: 3152
dataserviceversion: 2.0
sap-metadata-last-modified: Thu, 14 May 2020 23:58:07 GMT
cache-control: no-store, no-cache
sap-processing-info: ODataBEP=,crp=,RAL=,st=,MedCacheHub=SHM,codeployed=X,softstate=
sap-server: true
sap-perf-fesrec: 243070.000000
I tried setting the VersionIdentifier to a meaningless value in my test code (2nd update).
The update process seems to be successful, although the request header now has "If-Match" added to it.
(I was expecting the update to fail because the values never match, so I was hoping the update would fail.)
[2nd Update(setVersionIdenfifier)]
(It has masked some of the values.)
PATCH http://xxx.xxx.xxx.xxx:xxxxxx/sap/opu/odata/sap/API_BUSINESS_PARTNER/A_BusinessPartner(BusinessPartner='1000001') HTTP/1.1
x-csrf-token: xxx
Content-Type: application/json
Accept: application/json
If-Match: dummy
Authorization: Basic xxx
SAP-Connectivity-SCC-Location_ID: xxx
Proxy-Authorization: Bearer xxx
sap-language: en
sap-client: xxx
Content-Length: 55
If ETags are not part of OData service responses, then you should approach the IT/administrators who maintain the S/4 backend. The SAP Cloud SDK is only consuming the OData service. Unfortunately it can't leverage ETag support if it's disabled.
Have a simple get Customer api that's returning list of customers fine.
Setting up for service to service authentication, if I make this [Authenticated] and try to implement using ApiKeyAuthProvider, the req.GetApiKey returns null and I get an error;
Microsoft.AspNetCore.Hosting.Internal.WebHost:Information: Request starting HTTP/1.1 POST https://localhost:44347/api/customers application/json 0
Microsoft.AspNetCore.Hosting.Internal.WebHost:2019-07-01 16:50:34,004 [16] INFO Microsoft.AspNetCore.Hosting.Internal.WebHost - Request starting HTTP/1.1 POST https://localhost:44347/api/customers application/json 0
The thread 0x42cc has exited with code 0 (0x0).
The thread 0x302c has exited with code 0 (0x0).
ServiceStack.ServiceStackHost:2019-07-01 17:01:14,601 [16] ERROR ServiceStack.ServiceStackHost - ServiceBase<TRequest>::Service Exception
System.ArgumentOutOfRangeException: Length cannot be less than zero.
Parameter name: length
at System.String.Substring(Int32 startIndex, Int32 length)
at ServiceStack.Host.HttpRequestAuthentication.GetBasicAuth(IRequest httpReq) in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Host\HttpRequestAuthentication.cs:line 45
at ServiceStack.Host.HttpRequestAuthentication.GetBasicAuthUserAndPassword(IRequest httpReq) in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Host\HttpRequestAuthentication.cs:line 50
at ServiceStack.Auth.ApiKeyAuthProvider.PreAuthenticate(IRequest req, IResponse res) in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Auth\ApiKeyAuthProvider.cs:line 232
at ServiceStack.AuthenticateAttribute.PreAuthenticate(IRequest req, IEnumerable`1 authProviders) in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\AuthenticateAttribute.cs:line 96
at ServiceStack.AuthenticateAttribute.ExecuteAsync(IRequest req, IResponse res, Object requestDto) in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\AuthenticateAttribute.cs:line 74
at ServiceStack.Host.ServiceRunner`1.ExecuteAsync(IRequest req, Object instance, TRequest requestDto) in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Host\ServiceRunner.cs:line 127
Microsoft.AspNetCore.Hosting.Internal.WebHost:Information: Request finished in 640574.8754ms 400 application/json; charset=utf-8
Microsoft.AspNetCore.Hosting.Internal.WebHost:2019-07-01 17:01:14,607 [16] INFO Microsoft.AspNetCore.Hosting.Internal.WebHost - Request finished in 640574.8754ms 400 application/json; charset=utf-8
Clearly I have missed something obvious...any pointers appreciated.
// Register ORMLite connection
container.Register<IDbConnectionFactory>(dbFactory);
//Tell ServiceStack you want to persist User Auth Info in SQL Server
container.Register<IAuthRepository>(c => new OrmLiteAuthRepository(dbFactory));
// See https://docs.servicestack.net/api-key-authprovider
Plugins.Add(new AuthFeature(() => new AuthUserSession(),
new IAuthProvider[] {
new ApiKeyAuthProvider(AppSettings) {
SessionCacheDuration = TimeSpan.FromMinutes(10),
AllowInHttpParams = true, // Whether to allow API Keys in 'apikey' QueryString or FormData (e.g. `?apikey={APIKEY}`)
RequireSecureConnection = true,
},
}
) {
IncludeRegistrationService = true,
});
GlobalRequestFilters.Add((req, res, dto) =>
{
LastApiKey = req.GetApiKey();
});
Request
POST https://localhost:44347/api/customers HTTP/1.1
Host: localhost:44347
Connection: keep-alive
Content-Length: 2
Accept: application/json
Origin: https://localhost:44347
Authorization: yDOr26HsxyhpuRB3qbG07qfCmDhqutnA-yDOr26HsxyhpuRB3qbG07qfCmDhqutnA-yDOr26HsxyhpuRB3qbG07qfCmDhqutnA
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36
Content-Type: application/json
Referer: https://localhost:44347/swagger-ui/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en;q=0.9,en-US;q=0.8
{}
Response
HTTP/1.1 400 ArgumentOutOfRangeException
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8
Vary: Accept,Origin
Server: Microsoft-IIS/10.0
X-Powered-By: ServiceStack/5.50 NetCore/Windows
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, PATCH, OPTIONS
Access-Control-Allow-Headers: Content-Type
X-Startup-Errors: 1
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: Content-Disposition
X-SourceFiles: =?UTF-8?B?QzpcUmVwb3NcTUJXZWJccnZhcGlcUnZXZWJcUnZBcGlcYXBpXGN1c3RvbWVycw==?=
X-Powered-By: ASP.NET
Date: Wed, 03 Jul 2019 08:07:40 GMT
13e
{"responseStatus":{"errorCode":"ArgumentOutOfRangeException","message":"Length cannot be less than zero.\r\nParameter name: length","errors":[{"errorCode":"ArgumentOutOfRangeException","fieldName":"length","message":"Length cannot be less than zero.\r\n"}]},"responseCreatedUtcDateTime":"2019-07-03T08:07:40.7955827Z"}
0
Your client is sending an invalid Authorization Bearer Token, it needs to have the Authroization Bearer Token format:
Authorization: Bearer {Token}
If you're sending an Authenticated API Key or JWT Request via Open API it needs to have the Bearer prefix as per the Open API docs:
OK I had manually created a User and APIKey in the underlying tables and had used a UserAuthId 'SomeAuthId' i.e. letter in them, and the ORM repository code is expecting these to be integers. Its cool that I can see the code in github and debug this myself - thanks for the comment as it got me thinking and looking into my Auth setup.
sendFile is for sending files and it also figures out some interesting headers from the file (like content length). For a HEAD request I would ideally want the exact same headers but just skip the body.
There doesn't seem to be an option for this in the API. Maybe I can override something in the response object to stop it from sending anything?
Here's what I got:
res.sendFile(file, { headers: hdrs, lastModified: false, etag: false })
Has anyone solved this?
As Robert Klep has already written, the sendFile already has the required behavior of sending the headers and not sending the body if the request method is HEAD.
In addition to that, Express already handles HEAD requests for routes that have GET handlers defined. So you don't even need to define any HEAD handler explicitly.
Example:
let app = require('express')();
let file = __filename;
let hdrs = {'X-Custom-Header': '123'};
app.get('/file', (req, res) => {
res.sendFile(file, { headers: hdrs, lastModified: false, etag: false });
});
app.listen(3322, () => console.log('Listening on 3322'));
This sends its own source code on GET /file as can be demonstrated with:
$ curl -v -X GET localhost:3322/file
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 3322 (#0)
> GET /file HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:3322
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< X-Custom-Header: 123
< Accept-Ranges: bytes
< Cache-Control: public, max-age=0
< Content-Type: application/javascript
< Content-Length: 267
< Date: Tue, 11 Apr 2017 10:45:36 GMT
< Connection: keep-alive
<
[...]
The [...] is the body that was not included here.
Without adding any new handler this will also work:
$ curl -v -X HEAD localhost:3322/file
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 3322 (#0)
> HEAD /file HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:3322
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< X-Custom-Header: 123
< Accept-Ranges: bytes
< Cache-Control: public, max-age=0
< Content-Type: application/javascript
< Content-Length: 267
< Date: Tue, 11 Apr 2017 10:46:29 GMT
< Connection: keep-alive
<
This is the same but with no body.
Express uses send to implement sendFile, which already does exactly what you want.
I have a problem on logstash where my workers jump to 100% CPU while activating 1 specific filter. Im currently using this input + filter (on which grok parses ok):
input {
redis {
key => "logs_services_dyna_customer.lnxdev01"
data_type => ['list']
db => 8
host => "kibana.test.local"
type => "D_Customer"
tags => ["d_local"]
add_field => {
server => "lnxdev01"
}
codec => "plain"
}
}
filter {
if [type] in ["D_Customer"] {
mutate {
gsub => ["message", "\r", ""]
}
grok {
patterns_dir => "/etc/logstash/patterns/"
match => ["message", "^Request Headers:\n(%{WORD:request_method} %{DATA:service_url} %{DATA}\n)?(Host: %{HOSTNAME:request_host}\n)?(Connection: %{WORD:request_connection}\n)?(User-Agent: %{DATA:user_agent}\n)?(Content-Type: %{DATA:content_type}\n)?(SOAPAction: \"%{DATA:request_action}\"\n)?(Content-Length: %{NUMBER:request_content_length}\n)?(Cookie: %{DATA}\n)?(:?X-START-DATE: %{APACHE_DATETIME:request_start_date})?\n+Request Body:\n%{GREEDIERDATA:request_body}\n+Response Headers:\n(HTTP/%{DATA}(?<response_code>\d{3}) %{DATA:response_message}\n)?(Cache-Control: %{DATA}\n)?(Content-Length: %{NUMBER:response_content_length}\n)?(Content-Type: %{DATA:response_content_type}\n)?(Server: %{DATA:response_server}\n)?(X-AspNet-Version: %{DATA}\n)?(X-Powered-By: %{DATA}\n)?(Date: %{APACHE_DATETIME:response_date}\n)?(Connection: %{WORD:response_connection}\n)?\n+Response Body:\n%{GREEDIERDATA:response_body}$"]
add_field => ["category", "Service"]
}
date {
match => ["request_start_date", "EEE, dd MMMM YYYY HH:mm:ss z"]
}
date {
match => ["request_start_date", "EEE, dd MMMM YYYY HH:mm:ss z"]
target => "request_start_date"
}
date {
match => ["response_date", "EEE, dd MMMM YYYY HH:mm:ss z"]
target => "response_date"
}
}
}
And below the log itself:
Request Headers:
POST /Middleware.OIL.I01_SearchCustomer.Wcf/WcfService_Middleware_OIL_I01_SearchCustomer_App.svc HTTP/1.1
Host: w12vfdbiz101-t.dy.nl
Connection: close
User-Agent: PHP-SOAP/7.0.9-1+deb.sury.org~trusty+1
Content-Type: text/xml; charset=utf-8
SOAPAction: "SearchCustomer"
Content-Length: 1628
X-START-DATE: Mon, 07 Nov 2016 13:27:03 UTC
Request Body:
<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="http://www.dy.eu/Customer/V1-0"><SOAP-ENV:Body><v1:SearchCustomerRequest xmlns:v1="http://www.dy.eu/Customer/V1-0" xmlns:v2="http://dy/CoreComponents/Schemas/dy/V2-0" xmlns:urn="urn:oasis:names:specification:ubl:schema:xsd:CommonAggregateComponents-2" xmlns:urn1="urn:oasis:names:specification:ubl:schema:xsd:CommonBasicComponents-2" xmlns:v21="http://dy/CoreComponents/Schemas/dy/V2-0">
<v2:HeaderInfo>
<v2:TransactionId>58208127a1179</v2:TransactionId>
<v2:UserId>1</v2:UserId>
<v2:ApplicationId>Omnius</v2:ApplicationId>
<v2:PartyName>OGW</v2:PartyName>
</v2:HeaderInfo>
<v1:SearchedAccount>
<urn:PartyName>
<urn1:Name/>
</urn:PartyName>
</v1:SearchedAccount>
<v1:SearchedContact>
<v21:Role>
<v21:Person>
<urn1:ID/>
<urn1:FirstName>Friedrich</urn1:FirstName>
<urn1:FamilyName>Muller</urn1:FamilyName>
<urn:Contact>
<urn1:Telephone/>
<urn1:ElectronicMail/>
</urn:Contact>
<urn:ResidenceAddress>
<urn1:StreetName/>
<urn1:BuildingNumber/>
<urn1:CityName/>
<urn1:PostalZone/>
</urn:ResidenceAddress>
</v21:Person>
</v21:Role>
</v1:SearchedContact>
<v1:SearchedEquipment>
<urn:ManufacturersItemIdentification>
<urn1:ID/>
</urn:ManufacturersItemIdentification>
</v1:SearchedEquipment>
</v1:SearchCustomerRequest></SOAP-ENV:Body></SOAP-ENV:Envelope>
Response Headers:
HTTP/1.1 200 OK
Cache-Control: private
Content-Length: 2379
Content-Type: text/xml; charset=utf-8
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Mon, 07 Nov 2016 13:27:10 GMT
Connection: close
Response Body:
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"><s:Body><ns0:SearchCustomerResponse xmlns:ns0="http://www.dy.eu/Customer/V1-0" xmlns:ns1="http://uri.etsi.org/01903/v1.3.2#" xmlns:ns2="urn:oasis:names:specification:ubl:schema:xsd:CommonExtensionComponents-2" xmlns:ns3="urn:oasis:names:specification:ubl:schema:xsd:SignatureBasicComponents-2" xmlns:ns4="http://www.w3.org/2000/09/xmldsig#" xmlns:ns5="urn:oasis:names:specification:ubl:schema:xsd:QualifiedDataTypes-2" xmlns:ns6="urn:oasis:names:specification:ubl:schema:xsd:UnqualifiedDataTypes-2" xmlns:ns7="urn:un:unece:uncefact:data:specification:CoreComponentTypeSchemaModule:2" xmlns:ns8="urn:oasis:names:specification:ubl:schema:xsd:SignatureAggregateComponents-2" xmlns:ns9="http://uri.etsi.org/01903/v1.4.1#" xmlns:ns10="urn:oasis:names:specification:ubl:schema:xsd:CommonAggregateComponents-2" xmlns:ns11="http://Dy/CoreComponents/Schemas/Dy/V2-0" xmlns:ns12="urn:oasis:names:specification:ubl:schema:xsd:CommonSignatureComponents-2" xmlns:cbc="urn:oasis:names:specification:ubl:schema:xsd:CommonBasicComponents-2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><cbc:ReferenceID>8</cbc:ReferenceID><ns10:Status xsi:type="ns10:StatusType"><cbc:StatusReasonCode/><cbc:StatusReason/></ns10:Status><ns11:Success>true</ns11:Success><ns0:ReturnedNumberOfRecords>1</ns0:ReturnedNumberOfRecords><ns11:Account><ns10:PartyName><cbc:Name>CompanyName_0</cbc:Name></ns10:PartyName><ns10:PostalAddress xsi:type="ns10:AddressType"><cbc:StreetName>Street0</cbc:StreetName><cbc:BuildingNumber>1</cbc:BuildingNumber><cbc:CityName>City0</cbc:CityName><cbc:PostalZone>12345</cbc:PostalZone></ns10:PostalAddress><ns10:Contact xsi:type="ns10:ContactType"><cbc:ID>KS_12366</cbc:ID><cbc:ElectronicMail/></ns10:Contact><ns11:AccountCategory>KD</ns11:AccountCategory><ns11:AccountClass>P</ns11:AccountClass><ns11:ParentAccountNumber><cbc:ID>555</cbc:ID></ns11:ParentAccountNumber><ns11:BankAccountNumber>1234</ns11:BankAccountNumber><ns11:Role><ns11:Person><cbc:FirstName>Friedrich</cbc:FirstName><cbc:FamilyName>Müller</cbc:FamilyName><cbc:BirthDate>19700101</cbc:BirthDate><ns11:ContactPhoneNumber><ns11:PhoneNumber>0612387777</ns11:PhoneNumber><ns11:Type>Mobile</ns11:Type></ns11:ContactPhoneNumber></ns11:Person></ns11:Role></ns11:Account></ns0:SearchCustomerResponse></s:Body></s:Envelope>
So the question is, how come that logstash is spiking to 100% CPU and is this because of below filter? Should it be different or can it be more efficient?
Thanks
Pim
The issue was that somebody used GREEDIERDATA. This meant it will match on .* \n \r. If grok needs to parse it will get in a massive loop since it will not know when to stop at all and needs to iterate the complete entry all the time