I am trying to download some playlists off soundcloud and found a site that does this for you. Of course if the playlist is long, then it's super tedious to click each link to download. So I saved the HTML of the page and have parsed out the links. The idea is to use urllib or requests to download the files.
Here's my code:
opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
urllib.request.install_opener(opener)
track_url = 'https://scdownloader.io/download?track=zandex-hazerback-erox-stroke-bth-release&token=be1bc7997695495f756312886f566110'
track_name = 'BANG_THE_HOUSE___zandex-hazerback-erox-stroke-bth-release.mp3'
output_file = '/Users/ms/Desktop/playlist/{}'.format(track_name)
urllib.request.urlretrieve(track_url, output_file)
When I run the above code, it does save the file, but it arrives as a 1 byte file only.
I've tried other permutations using requests but basically either it doesn't work, downloads and saves a zero byte file, or does work to download and save a 1 byte file... just can't get the whole thing!
Also note, I have to send headers b/c otherwise I get a 403 error.
Any help is greatly appreciated!
Thank you!
EDIT:
Per the comments below, here's what the urlretrieve http response is:
Date: Fri, 15 Mar 2019 23:52:44 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: close
Set-Cookie: __cfduid=dcc5f95391fac83973cc77648c0e8c0391552693964; expires=Sat, 14-Mar-20 23:52:44 GMT; path=/; domain=.scdownloader.io; HttpOnly; Secure
X-Powered-By: PHP/5.6.36
Set-Cookie: PHPSESSID=fsnrrrtpnrav3vq5u2t9vfvrp7; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding,User-Agent
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 4b82671d38067790-LAX
Related
I want to integrate a bot via a weebhook between Gitlab and discord, so I've configured the bot first, copied his url and put it into the gitlab weebhook configuration input and set it for sending push updates to the Discord server.
With a real push test, I have (with the body)
Request headers:
Content-Type: application/json
X-Gitlab-Event: Push Hook
and as response
Response headers:
Date: Tue, 26 May 2020 18:46:48 GMT
Content-Type: application/json
Content-Length: 58
Connection: close
Set-Cookie: __cfduid=d374998c2f84e3e20b75bbdec88fb63d91590518808; expires=Thu, 25-Jun-20 18:46:48 GMT; path=/; domain=.discordapp.com; HttpOnly; SameSite=Lax, __cfruid=418f7199379a53d23012d37b15f2ac5a3aac36b6-1590518808; path=/; domain=.discordapp.com; HttpOnly; Secure; SameSite=None
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Ratelimit-Bucket: 3cd1f278bd0ecaf11e0d2391374c011d
X-Ratelimit-Limit: 5
X-Ratelimit-Remaining: 4
X-Ratelimit-Reset: 1590518811
X-Ratelimit-Reset-After: 2
X-Envoy-Upstream-Service-Time: 12
Via: 1.1 google
Cf-Cache-Status: DYNAMIC
Cf-Request-Id: 02f3e816a1000004823d920200000001
Expect-Ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
Cf-Ray: 5999a9376a790482-CDG
but got the error:
Response body:
{"message": "Cannot send an empty message", "code": 50006}
or also
Hook executed successfully but returned HTTP 400 {"message": "Cannot send an empty message", "code": 50006}
Thanks for help
You need to use the "Integrations" feature for "Discord notifications" instead of regular webhook.
See documentation here
I am trying to publish contact group into Azure Active Directory.
Steps that I followed:
Register Application in Azure and get the Tenant ID and Client ID.
Send the CFHTTP Request for token (I need help here)
Send token and other parameters (I need help here) to publish the contact groups
cfhttp(method="POST", charset="utf-8", url="https://login.microsoftonline.com/common/oauth2/v2.0/response_type=code", result="result") {
cfhttpparam(name="Content-Type", type="header", value="application/x-www-form-urlencoded");
cfhttpparam(name="client_id", type="formfield", value="28d5e75-a762-45a2-9504-e9bbaa3294af");
cfhttpparam(name="grant_type", type="formfield", value="authorization_code");
cfhttpparam(name="mimetype", type="header", value="application/xml");
}
Response:
Header HTTP/1.1 404 Not Found Cache-Control: private Strict-Transport-Security: max-age=31536000; includeSubDomains X-Content-Type-Options: nosniff x-ms-request-id: caeef2f8-71a0-45f6-8820-63d880f02700 x-ms-ests-server: 2.1.10244.21 - WST ProdSlices P3P: CP="DSP CUR OTPi IND OTRi ONL FIN" Set-Cookie: x-ms-gateway-slice=prod; path=/; SameSite=None; secure; HttpOnly Date: Mon, 23 Mar 2020 18:20:59 GMT Connection: close Content-Length: 0
Mimetype Unable to determine MIME type of file.
When i use the curl command all the headers are correct but files are not caching as
CF-Cache-Status: MISS
is shown by curl command .
> < HTTP/1.1 200 OK < Date: Sat, 02 Jun 2018 05:57:28 GMT <
> Content-Type: image/png < Content-Length: 2293 < Connection:
> keep-alive < Set-Cookie:
> __cfduid=d3cf179afd212660124585350d19e88c81527919048; expires=Sun, 02-Jun-19 05:57:28 GMT; path=/; domain=.mayan.cf; HttpOnly <
> X-Powered-By: Express < Accept-Ranges: bytes < Cache-Control: public,
> max-age=86400 < Last-Modified: Fri, 01 Jun 2018 15:39:06 GMT < ETag:
> W/"8f5-163bc009a0c" < CF-Cache-Status: MISS < Expires: Sun, 03 Jun
> 2018 05:57:28 GMT < Server: cloudflare < CF-RAY: 4247aec4300930f6-SIN
> < { [1905 bytes data]
> * Connection #0 to host mayan.cf left intact
Cloudflare uses some heuristics to try to guess, based on the request alone, whether the content is static (cacheable) or dynamic. If it guesses wrong at request time, it won't cache the response. You can disable this heuristic by setting the Page Rule "Cache Level" to "Cache Everything" -- then Cloudflare will assume that the response will always be cacheable.
Try setting that page rule and see if it helps.
If the Cache-Control header is set to "private", "no-store",
"no-cache", or "max-age=0", or if there is a cookie in the response,
then Cloudflare will not cache the resource.
https://support.cloudflare.com/hc/en-us/articles/202775670-How-Do-I-Tell-Cloudflare-What-to-Cache-
I'm trying to provide the correlation id for this issue but I'm not sure how to obtain it. I assume it should be in response, is that correct? Is it the client-request-id maybe?
HTTP Response
HTTP/1.1 403 Forbidden
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json;odata=minimalmetadata;streaming=true;charset=utf-8
Expires: -1
Server: Microsoft-IIS/8.5
ocp-aad-diagnostics-server-name: CGa9FFPZASZM+usTGvTzaOZ4zSMxhpbWh2mO9+eJZWs=
request-id: 6ddd7a86-02c9-4dcf-9932-b17afc976107
client-request-id: 50478d75-542d-4e0e-96d9-de913d632471
x-ms-dirapi-data-contract-version: 1.6
ocp-aad-session-key: some-really-long-value
X-Content-Type-Options: nosniff
DataServiceVersion: 3.0;
Strict-Transport-Security: max-age=31536000; includeSubDomains
Access-Control-Allow-Origin: *
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Duration: 782602
X-Powered-By: ASP.NET
Date: Wed, 09 Aug 2017 14:22:21 GMT
Content-Length: 139
{"odata.error":{"code":"Authorization_RequestDenied","message":{"lang":"en","value":"Insufficient privileges to complete the operation."}}}
Token Request Response
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Server: Microsoft-IIS/8.5
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
client-request-id: 8c500700-b75c-403a-a2c9-c6dfa27049d3
x-ms-request-id: 75627b63-fbd6-4fa8-95b5-b6a325d60900
P3P: CP="DSP CUR OTPi IND OTRi ONL FIN"
Set-Cookie: esctx=some-value; domain=.login.microsoftonline.com; path=/; secure; HttpOnly
Set-Cookie: x-ms-gateway-slice=006; path=/; secure; HttpOnly
Set-Cookie: stsservicecookie=ests; path=/; secure; HttpOnly
X-Powered-By: ASP.NET
Date: Wed, 09 Aug 2017 18:33:33 GMT
Content-Length: 1437
{"token_type":"Bearer","expires_in":"3599","ext_expires_in":"262800","expires_on":"1502307214","not_before":"1502303314","resource":"https://graph.windows.net/","access_token":"some-token-value"}
The correlation ID (also called client-request-id) can be found in interactions with the Azure AD service.
The request immediately before this when you were asking the Azure AD endpoints for tokens will include the ID in the response.
https://login.microsoftonline.com/...
https://login.windows.net/...
I have to send request using curl, in that request i needs to set the cookie from the below response.
So can some one help me how to pass all the cookies in the curl request
HTTP/1.1 200 OK
X-Powered-By: Express
Access-Control-Allow-Origin:undefined
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: GET,POST,OPTIONS
Access-Control-Allow-Headers: Content-Type
set-cookie: SESSIONID=agpuB2CqPDhkupxfbrZXCR; Path=/ABC
set-cookie: bac_cookie=7959aacd5cc48;Domain=abc.com;Expires=Mon, 05-Dec-2016 18:47:42 GMT; Path=/
set-cookie: beID=0023T21; Domain=abc.com; Expires=Mon,05-Dec-2016 18:47:42 GMT; Path=/
set-cookie:USER_JSESSIONID=agpuB2CqPDhkupxfbrJlbXF3;Domain=abc.com;Expires=Mon, 05-Dec-2016 18:47:42 GMT; Path=/
set-cookie:bac_user_info="UserName|UserLName|UserMailId"; Version=1;Domain=abc.com; Max-Age=3600; Path=/
set-cookie: beID=0023T21; Domain=abc.com; Expires=Mon, 05-Dec-2016 18:47:42 GMT; Path=/
set-cookie:NSC_WB2-CF-8080=ffffffff09ae530a45525d5f4f58455e445a4a4229a0;path=/;httponly
Content-Type: application/json; charset=utf-8
Content-Length: 131
ETag: W/"83-+Jfk31hxbwmLWd1hTlIO1Q"
Date: Mon, 05 Dec 2016 17:47:42 GMT
Connection: abc
If you want to send multiple cookies, just sepreate them by ;
curl -v --cookie "TOKEN1=Yes;TOKEN2=no" http://127.0.0.1:5000