Upload file using Curl on linux bash not working - linux

I am trying from last 15-20 days but its not getting sucs. Please check below, if any sharepoint side security need to change or any command modification needed. please help ..Please help
I am trying to upload a file from Linux to SharePoint with my SharePoint login credentials. I use the cURL utility to achieve this.It's not getting successful ..
I tried to download file from sharepoint and its downloading file very easly using below command
curl -k --ntlm -u domain/username:password -O https://sharepointserver.com/sites/mysite/myfile.txt
The Upload command used is :
curl ---ntlm -u domain/username:password --upload-file myfile.txt -k "https://sharepointserver.com/sites/mysite
Output using verbose on:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to connect() to proxy 'proxy server' port 80 (#0)
* Trying 'Ip address'...
* Connected to proxy server (proxy Ip) port 80 (#0)
* Establish HTTP proxy tunnel to test.sharepoint.com:443
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* Server auth using NTLM with user 'domain/user'
> CONNECT test.sharepoint.com:443 HTTP/1.1
> Host: test.sharepoint.com:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
>
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0< HTTP/1.0 200 Connection established
< Set-Cookie: BIGipServerSquid_Dev_Pool=rd2319o0000000000000000000033441018o80; path=/; Httponly
<
* Proxy replied OK to CONNECT request
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=*.sharepoint.com,OU=Microsoft Corporation,O=Microsoft Corporation,L=Redmond,ST=WA,C=US
* start date: Mar 07 21:35:03 2018 GMT
* expire date: Mar 06 21:35:03 2020 GMT
* common name: *.sharepoint.com
* issuer: CN=Microsoft IT TLS CA 1,OU=Microsoft IT,O=Microsoft Corporation,L=Redmond,ST=Washington,C=US
* Server auth using NTLM with user 'domain/user'
> GET /:x:/r/sites/TheHub/ECP/Project%20Glide/Project%20Glide%20Collaboration%20site/Shared%20Documents/14.%20Document%20and%20Records%20Management/TWS%20Reports/Development HTTP/1.1
> Authorization: NTLM TlRMTVNTUAABAAAABoIIAAAAAAAAAAAAAAAAAAAAAAA=
> User-Agent: curl/7.29.0
> Host: test.sharepoint.com
> Accept: */*
>
**< HTTP/1.1 401 Unauthorized**
< P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
< SPRequestGuid: de7b2f9f-a041-2000-ccc6-0abdd166adb5
< request-id: de7b2f9f-a041-2000-ccc6-0abdd166adb5
< MS-CV: ny973kGgACDMxgq90WattQ.0
< Strict-Transport-Security: max-age=31536000
< X-FRAME-OPTIONS: SAMEORIGIN
< SPRequestDuration: 5
< SPIisLatency: 5
< X-Powered-By: ASP.NET
< MicrosoftSharePointTeamServices: 16.0.0.19708
< X-Content-Type-Options: nosniff
< X-MS-InvokeApp: 1; RequireReadOnly
< X-MSEdge-Ref: Ref A: 20DC3927B781464DAF54406EC7F8A4AB Ref B: LON21EDGE0516 Ref C: 2020-01-27T05:54:46Z
< Date: Mon, 27 Jan 2020 05:54:45 GMT
< Content-Length: 0
<
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Connection #0 to host "proxyserver address" left intact
and If I am added -o(small o) then using basic authentication and command is getting successful but the file is not getting upload ... below is output for above command with small o(-o)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to connect() to proxy 'proxy server' port 80 (#0)
* Trying Ip address...
* Connected to 'proxy server' (IP address) port 80 (#0)
* Establish HTTP proxy tunnel to test.sharepoint.com:443
* Server auth using Basic with user 'domain/user'
> CONNECT test.sharepoint.com:443 HTTP/1.1
> Host: test.sharepoint.com:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.0 200 Connection established
< Set-Cookie: BIGipServerSquid_Dev_Pool=rd2319o00000000000000000000ffff0add1018o80; path=/; Httponly
<
* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=*.sharepoint.com,OU=Microsoft Corporation,O=Microsoft Corporation,L=Redmond,ST=WA,C=US
* start date: Mar 07 21:35:03 2018 GMT
* expire date: Mar 06 21:35:03 2020 GMT
* common name: *.sharepoint.com
* issuer: CN=Microsoft IT TLS CA 1,OU=Microsoft IT,O=Microsoft Corporation,L=Redmond,ST=Washington,C=US
* Server auth using Basic with user 'domain/user'
> PUT /:x:/r/sites/TheHub/ECP/Project%20Glide/Project%20Glide%20Collaboration%20site/Shared%20Documents/14.%20Document%20and%20Records%20Management/TWS%20Reports/Development HTTP/1.1
> Authorization: Basic Z2JidXBhZ3JvdXAvc2F3YW50bjozbWF0b231ml2QVNAQEBA
> User-Agent: curl/7.29.0
> Host: test.sharepoint.com
> Accept: */*
> Content-Length: 311
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
} [data not shown]
* We are completely uploaded and fine
< HTTP/1.1 301 Moved Permanently
< Content-Type: text/plain
< Location: https://test.sharepoint.com/sites/TheHub/ECP/Project%20Glide/Project%20Glide%20Collaboration%20site/Shared%20Documents/14.%20Document%20and%20Records%20Management/TWS%20Reports/Development?cid=50561376-1bb4-4f92-be7c-0a710fb78cc3
< P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
< SPRequestGuid: 857c2f9f-9013-2000-6cd2-3c84489a0ad4
< request-id: 857c2f9f-9013-2000-6cd2-3c84489a0ad4
< MS-CV: ny98hROQACBs0jyESJoK1A.0
< Strict-Transport-Security: max-age=31536000
< SPRequestDuration: 5
< SPIisLatency: 5
< X-Powered-By: ASP.NET
< MicrosoftSharePointTeamServices: 16.0.0.19708
< X-Content-Type-Options: nosniff
< X-MS-InvokeApp: 1; RequireReadOnly
< X-MSEdge-Ref: Ref A: 49395CFB685F40A184BEA4E5406D6D64 Ref B: LON21EDGE1013 Ref C: 2020-01-27T06:06:09Z
< Date: Mon, 27 Jan 2020 06:06:08 GMT
< Content-Length: 0
<
100 311 0 0 100 311 0 1323 --:--:-- --:--:-- --:--:-- 1329

Related

Can't open remote ssh directory after update

I have been using VSCode on a Windows laptop for an ssh remote directory on a Linux machine. Things were working fine until an update of VSCode 1.68.0 and now I can't open the remote directory.
On Windows, the output says:
[12:55:24.183] Log Level: 2
[12:55:24.185] remote-ssh#0.82.1
[12:55:24.185] win32 x64
[12:55:24.226] SSH Resolver called for "ssh-remote+ubuntu", attempt 1
[12:55:24.227] "remote.SSH.useLocalServer": false
[12:55:24.227] "remote.SSH.showLoginTerminal": false
[12:55:24.227] "remote.SSH.remotePlatform": {"runon":"linux","pfuntner-kube":"linux","everyday":"linux","kube":"linux","ubuntu":"linux"}
[12:55:24.227] "remote.SSH.path": C:\cygwin64\bin\ssh.exe
[12:55:24.228] "remote.SSH.configFile": c:\users\jpfuntne\.ssh\config
[12:55:24.228] "remote.SSH.useFlock": true
[12:55:24.228] "remote.SSH.lockfilesInTmp": false
[12:55:24.228] "remote.SSH.localServerDownload": auto
[12:55:24.229] "remote.SSH.remoteServerListenOnSocket": false
[12:55:24.229] "remote.SSH.showLoginTerminal": false
[12:55:24.229] "remote.SSH.defaultExtensions": []
[12:55:24.229] "remote.SSH.loglevel": 2
[12:55:24.229] "remote.SSH.enableDynamicForwarding": true
[12:55:24.230] "remote.SSH.enableRemoteCommand": false
[12:55:24.230] "remote.SSH.serverPickPortsFromRange": {}
[12:55:24.230] "remote.SSH.serverInstallPath": {}
[12:55:24.239] SSH Resolver called for host: ubuntu
[12:55:24.239] Setting up SSH remote "ubuntu"
[12:55:24.247] Using commit id "4af164ea3a06f701fe3e89a2bcbb421d2026b68f" and quality "stable" for server
[12:55:24.254] Install and start server if needed
[12:55:24.257] Checking ssh with "C:\cygwin64\bin\ssh.exe -V"
[12:55:24.568] > OpenSSH_8.8p1, OpenSSL 1.1.1l 24 Aug 2021
[12:55:24.574] Using SSH config file "c:\users\jpfuntne\.ssh\config"
[12:55:24.575] Running script with connection command: "C:\cygwin64\bin\ssh.exe" -T -D 65341 -F "c:\users\jpfuntne\.ssh\config" ubuntu bash
[12:55:24.591] Terminal shell path: C:\WINDOWS\System32\cmd.exe
[12:55:25.421] > WARNING!!!
> READ THIS BEFORE ATTEMPTING TO LOGON
>
> This System is for the use of authorized users only. Individuals
> using this computer without authority, or in excess of their
> authority, are subject to having all of their activities on this
> system monitored and recorded by system personnel. In the course
> of monitoring individuals improperly using this system, or in the
> course of system maintenance, the activities of authorized users
> may also be monitored. Anyone using this system expressly
> consents to such monitoring and is advised that if such
> monitoring reveals possible criminal activity, system personnel
> may provide the evidence of such monitoring to law enforcement
> officials.
>
> Cisco Acceptable Use Policy:
> http://wwwin.cisco.com/c/cec/organizations/security-trust/infosec/policies
> .html]0;C:\WINDOWS\System32\cmd.exe
[12:55:25.421] Got some output, clearing connection timeout
[12:55:26.367] > 28e76cc256c4: running
[12:55:26.405] > Acquiring lock on /home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb42
> 1d2026b68f/vscode-remote-lock.ubuntu.4af164ea3a06f701fe3e89a2bcbb421d2026b68f
[12:55:26.430] > Installing to /home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d20
> 26b68f...
> 28e76cc256c4%%1%%
> Downloading with wget
[12:55:33.759] > wget download failed
> failed: Connection timed out.
> printenv:
> SHELL=/bin/bash
> PWD=/home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026b68f
> LOGNAME=ubuntu
> XDG_SESSION_TYPE=tty
> MOTD_SHOWN=pam
> HOME=/home/ubuntu
> LANG=C.UTF-8
> VSCODE_AGENT_FOLDER=/home/ubuntu/.vscode-server
> SSH_CONNECTION=10.82.235.230 65347 64.102.189.72 22
> XDG_SESSION_CLASS=user
> USER=ubuntu
> SHLVL=0
> XDG_SESSION_ID=5
> XDG_RUNTIME_DIR=/run/user/1000
> SSH_CLIENT=10.82.235.230 65347 22
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
> :/usr/local/games:/snap/bin
> DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
> OLDPWD=/home/ubuntu
> _=/usr/bin/printenv
[12:55:33.764] >
> Trigger local server download
> 28e76cc256c4:trigger_server_download
> platform==linux==
> vscodeArch==x64==
> destFolder==/home/ubuntu/.vscode-server/bin/==
> destFolder2==4af164ea3a06f701fe3e89a2bcbb421d2026b68f==
> 28e76cc256c4:trigger_server_download_end
> Waiting for client to transfer server archive...
> Waiting for /home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026
> b68f/vscode-scp-done.flag and vscode-server.tar.gz to exist
[12:55:33.765] Got request to download on client for {"platform":"linux","arch":"x64","destFolder":"/home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026b68f"}
[12:55:33.765] Downloading VS Code server locally...
[12:55:33.770] >
>
[12:55:34.131] Resolver error: Error: Server returned 404
at v (vscode-file://vscode-app/c:/Users/jpfuntne/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/workbench/workbench.desktop.main.js:622:38476)
at k.download (vscode-file://vscode-app/c:/Users/jpfuntne/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/workbench/workbench.desktop.main.js:623:1592)
[12:55:34.136] ------
On Linux, all I have is:
$ find .vscode-server -type f | xargs -r ls -ld
-rw-rw-r-- 1 ubuntu ubuntu 0 Jun 15 12:55 .vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026b68f/vscode-remote-lock.ubuntu.4af164ea3a06f701fe3e89a2bcbb421d2026b68f
-rw-rw-r-- 1 ubuntu ubuntu 0 Jun 15 12:55 .vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026b68f/vscode-server.tar.gz
$
When I was first having trouble, I did rm -rf .vscode to fix things but it didn't help.
Can anyone help?
I tried to start VSCode again today. At first I thought it had the same problem but it looks like things are working ok now and I got in!
[07:14:44.009] Log Level: 2
[07:14:44.010] remote-ssh#0.82.1
[07:14:44.010] win32 x64
[07:14:44.048] SSH Resolver called for "ssh-remote+ubuntu", attempt 1
[07:14:44.048] "remote.SSH.useLocalServer": false
[07:14:44.049] "remote.SSH.showLoginTerminal": false
[07:14:44.049] "remote.SSH.remotePlatform": {"runon":"linux","pfuntner-kube":"linux","everyday":"linux","kube":"linux","ubuntu":"linux"}
[07:14:44.049] "remote.SSH.path": C:\cygwin64\bin\ssh.exe
[07:14:44.049] "remote.SSH.configFile": c:\users\jpfuntne\.ssh\config
[07:14:44.049] "remote.SSH.useFlock": true
[07:14:44.049] "remote.SSH.lockfilesInTmp": false
[07:14:44.049] "remote.SSH.localServerDownload": auto
[07:14:44.050] "remote.SSH.remoteServerListenOnSocket": false
[07:14:44.050] "remote.SSH.showLoginTerminal": false
[07:14:44.050] "remote.SSH.defaultExtensions": []
[07:14:44.050] "remote.SSH.loglevel": 2
[07:14:44.050] "remote.SSH.enableDynamicForwarding": true
[07:14:44.050] "remote.SSH.enableRemoteCommand": false
[07:14:44.050] "remote.SSH.serverPickPortsFromRange": {}
[07:14:44.050] "remote.SSH.serverInstallPath": {}
[07:14:44.056] SSH Resolver called for host: ubuntu
[07:14:44.056] Setting up SSH remote "ubuntu"
[07:14:44.062] Using commit id "4af164ea3a06f701fe3e89a2bcbb421d2026b68f" and quality "stable" for server
[07:14:44.068] Install and start server if needed
[07:14:44.071] Checking ssh with "C:\cygwin64\bin\ssh.exe -V"
[07:14:44.354] > OpenSSH_8.8p1, OpenSSL 1.1.1l 24 Aug 2021
[07:14:44.386] Using SSH config file "c:\users\jpfuntne\.ssh\config"
[07:14:44.387] Running script with connection command: "C:\cygwin64\bin\ssh.exe" -T -D 63895 -F "c:\users\jpfuntne\.ssh\config" ubuntu bash
[07:14:44.400] Terminal shell path: C:\WINDOWS\System32\cmd.exe
[07:14:46.375] > WARNING!!!
> READ THIS BEFORE ATTEMPTING TO LOGON
>
> This System is for the use of authorized users only. Individuals
> using this computer without authority, or in excess of their
> authority, are subject to having all of their activities on this
> system monitored and recorded by system personnel. In the course
> of monitoring individuals improperly using this system, or in the
> course of system maintenance, the activities of authorized users
> may also be monitored. Anyone using this system expressly
> consents to such monitoring and is advised that if such
> monitoring reveals possible criminal activity, system personnel
> may provide the evidence of such monitoring to law enforcement
> officials.
>
> Cisco Acceptable Use Policy:
> http://wwwin.cisco.com/c/cec/organizations/security-trust/infosec/policies
> .html]0;C:\WINDOWS\System32\cmd.exe
[07:14:46.375] Got some output, clearing connection timeout
[07:14:47.361] > cb747be286c8: running
> Acquiring lock on /home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb42
> 1d2026b68f/vscode-remote-lock.ubuntu.4af164ea3a06f701fe3e89a2bcbb421d2026b68f
> Installing to /home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d20
> 26b68f...
> cb747be286c8%%1%%
> Downloading with wget
[07:14:54.685] > wget download failed
> failed: Connection timed out.
> printenv:
> SHELL=/bin/bash
[07:14:54.692] >
> PWD=/home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026b68f
> LOGNAME=ubuntu
> XDG_SESSION_TYPE=tty
> MOTD_SHOWN=pam
> HOME=/home/ubuntu
> LANG=C.UTF-8
> VSCODE_AGENT_FOLDER=/home/ubuntu/.vscode-server
> SSH_CONNECTION=10.82.240.153 63898 64.102.189.72 22
> XDG_SESSION_CLASS=user
> USER=ubuntu
> SHLVL=0
> XDG_SESSION_ID=63
> XDG_RUNTIME_DIR=/run/user/1000
> SSH_CLIENT=10.82.240.153 63898 22
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
> :/usr/local/games:/snap/bin
> DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
> OLDPWD=/home/ubuntu
> _=/usr/bin/printenv
[07:14:54.695] >
> Trigger local server download
> cb747be286c8:trigger_server_download
> platform==linux==
> vscodeArch==x64==
> destFolder==/home/ubuntu/.vscode-server/bin/==
> destFolder2==4af164ea3a06f701fe3e89a2bcbb421d2026b68f==
> cb747be286c8:trigger_server_download_end
> Waiting for client to transfer server archive...
> Waiting for /home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026
> b68f/vscode-scp-done.flag and vscode-server.tar.gz to exist
>
[07:14:54.697] Got request to download on client for {"platform":"linux","arch":"x64","destFolder":"/home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026b68f"}
[07:14:54.698] Downloading VS Code server locally...
[07:14:58.248] Downloaded VS Code server to c:\Users\jpfuntne\AppData\Local\Temp\03ba8324-01be-4f4c-bd89-538da94836ae
[07:14:58.253] Renamed VS Code server to c:\Users\jpfuntne\AppData\Local\Temp\vscode_server_1655378098249\vscode-server.tar.gz
[07:14:58.257] Checking ssh with "C:\cygwin64\bin\ssh.exe -V"
[07:14:58.555] > OpenSSH_8.8p1, OpenSSL 1.1.1l 24 Aug 2021
[07:14:58.581] Expecting scp to be next to ssh at C:\cygwin64\bin\scp.exe
[07:14:58.582] Testing scp with "C:\cygwin64\bin\scp.exe"
[07:14:59.442] scp exited with code: 1
[07:14:59.442] Got stderr from scp: usage: scp [-346ABCOpqRrsTv] [-c cipher] [-D sftp_server_path] [-F ssh_config]
[-i identity_file] [-J destination] [-l limit]
[-o ssh_option] [-P port] [-S program] source ... target
[07:14:59.443] Using SSH config file "c:\users\jpfuntne\.ssh\config"
[07:14:59.445] Copying file to remote with "C:\cygwin64\bin\scp.exe" -F "c:\users\jpfuntne\.ssh\config" "vscode-server.tar.gz" "vscode-scp-done.flag" ubuntu:"/home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026b68f"
[07:14:59.447] Using cwd: file:///c%3A/Users/jpfuntne/AppData/Local/Temp/vscode_server_1655378098249
[07:14:59.447] Terminal shell path: C:\WINDOWS\System32\cmd.exe
[07:15:00.744] > WARNING!!!
> READ THIS BEFORE ATTEMPTING TO LOGON
>
> This System is for the use of authorized users only. Individuals
> using this computer without authority, or in excess of their
> authority, are subject to having all of their activities on this
> system monitored and recorded by system personnel. In the course
> of monitoring individuals improperly using this system, or in the
> course of system maintenance, the activities of authorized users
> may also be monitored. Anyone using this system expressly
> consents to such monitoring and is advised that if such
> monitoring reveals possible criminal activity, system personnel
> may provide the evidence of such monitoring to law enforcement
> officials.
>
> Cisco Acceptable Use Policy:
> http://wwwin.cisco.com/c/cec/organizations/security-trust/infosec/policies
> .html]0;C:\WINDOWS\System32\cmd.exe
[07:15:01.620] > vscode-server.tar.gz 0% 0 0.0KB/s --:-- ETA
[07:15:02.630] > vscode-server.tar.gz 32% 17MB 17.0MB/s 00:02 ETA
[07:15:03.645] > vscode-server.tar.gz 66% 36MB 17.1MB/s 00:01 ETA
[07:15:04.691] > vscode-server.tar.gz 100% 54MB 17.4MB/s 00:03
[07:15:04.705] >
[07:15:04.722] > vscode-scp-done.flag 0% 0 0.0KB/s --:-- ETA
[07:15:04.736] > vscode-scp-done.flag 100% 9 0.7KB/s 00:00
[07:15:06.150] "Copy server to host" terminal command done
[07:15:06.702] > Found flag and server on host
> cb747be286c8%%2%%
> tar --version:
[07:15:06.717] >
> tar (GNU tar) 1.30
> Copyright (C) 2017 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
>
> Written by John Gilmore and Jay Fenlason.
[07:15:08.979] > Checking /home/ubuntu/.vscode-server/.4af164ea3a06f701fe3e89a2bcbb421d2026b68f.l
> og and /home/ubuntu/.vscode-server/.4af164ea3a06f701fe3e89a2bcbb421d2026b68f.pid
> for a running server
[07:15:09.019] > Starting server with command... /home/ubuntu/.vscode-server/bin/4af164ea3a06f701
> fe3e89a2bcbb421d2026b68f/bin/code-server --start-server --host=127.0.0.1 --accep
> t-server-license-terms --enable-remote-auto-shutdown --port=0 --telemetry-level
> all &> "/home/ubuntu/.vscode-server/.4af164ea3a06f701fe3e89a2bcbb421d2026b68f.
> log" < /dev/null
> printenv:
> SHELL=/bin/bash
> PWD=/home/ubuntu
> LOGNAME=ubuntu
> XDG_SESSION_TYPE=tty
> MOTD_SHOWN=pam
> HOME=/home/ubuntu
> LANG=C.UTF-8
> VSCODE_AGENT_FOLDER=/home/ubuntu/.vscode-server
> SSH_CONNECTION=10.82.240.153 63898 64.102.189.72 22
> XDG_SESSION_CLASS=user
> USER=ubuntu
> SHLVL=0
> XDG_SESSION_ID=63
> XDG_RUNTIME_DIR=/run/user/1000
> SSH_CLIENT=10.82.240.153 63898 22
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
> :/usr/local/games:/snap/bin
> DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
> OLDPWD=/home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026b
> 68f
> _=/usr/bin/printenv
> rm: cannot remove '/home/ubuntu/.vscode-server/.4af164ea3a06f701fe3e89a2bcbb421d
> 2026b68f.token': No such file or directory
[07:15:09.029] >
> Spawned remote server: 60287
[07:15:09.036] >
> Waiting for server log...
[07:15:09.052] >
[07:15:09.080] > Waiting for server log...
[07:15:09.146] >
> *
> * Reminder: You may only use this software with Visual Studio family products,
> * as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057)
> *
>
[07:15:09.171] > cb747be286c8: start
> SSH_AUTH_SOCK====
> DISPLAY====
> webUiAccessToken====
> listeningOn==35689==
> osReleaseId==ubuntu==
> arch==x86_64==
> tmpDir==/run/user/1000==
> platform==linux==
> unpackResult==success==
> didLocalDownload==1==
> downloadTime====
> installTime==2258==
> extInstallTime====
> serverStartTime==122==
> connectionToken==1a1a1a1a-1a1a-11a1-aa1a-1111a1111a1a==
[07:15:09.184] > cb747be286c8: end
[07:15:09.184] Received install output:
SSH_AUTH_SOCK====
DISPLAY====
webUiAccessToken====
listeningOn==35689==
osReleaseId==ubuntu==
arch==x86_64==
tmpDir==/run/user/1000==
platform==linux==
unpackResult==success==
didLocalDownload==1==
downloadTime====
installTime==2258==
extInstallTime====
serverStartTime==122==
connectionToken==1a1a1a1a-1a1a-11a1-aa1a-1111a1111a1a==
[07:15:09.185] Remote server is listening on 35689
[07:15:09.185] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":35689},"osReleaseId":"ubuntu","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","display":"","tmpDir":"/run/user/1000","platform":"linux","connectionToken":"1a1a1a1a-1a1a-11a1-aa1a-1111a1111a1a"},"installTime":2258,"serverStartTime":122,"didLocalDownload":true,"installUnpackCode":"success"}
[07:15:09.193] Starting forwarding server. localPort 63909 -> socksPort 63895 -> remotePort 35689
[07:15:09.195] Forwarding server listening on 63909
[07:15:09.195] Waiting for ssh tunnel to be ready
[07:15:09.257] [Forwarding server 63909] Got connection 0
[07:15:09.260] Tunneled 35689 to local port 63909
[07:15:09.260] Resolved "ssh-remote+ubuntu" to "127.0.0.1:63909"
[07:15:09.272] ------
[07:15:09.300] [Forwarding server 63909] Got connection 1
[07:15:09.650] [Forwarding server 63909] Got connection 2
On Linux:
$ find .vscode-server -type f | wc
1798 1798 196423
$

Get http code when uploading file using curl and printing curl output

I'm writing a bash script where I want to upload a file using curl. I want to print in a log file the output from stdout and stderr (to see if the upload is being completed). This is what works so far:
curl -T home/user/mydir/filename <my url>/.../filename >> home/user/mydir/script.log 2>&1
Now, I also want to get the returned http code so I can use it somewhere else in my script. The following works as well:
CODE=$(curl -w '%{http_code}' -T home/user/mydir/filename <my url>/.../filename)
How can I combine these two commands? I want to print the stdout and stderr in the log file (append, to be precise) and also store the returned http code in a variable for later use.
Thank you.
Can you do something like this (the first < line is the return code) (from the verbose option). The -o can be - (stdout) OR any other file you want??
# curl -v -o /dev/null http://www.google.com
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 142.250.76.100...
* TCP_NODELAY set
* Connected to www.google.com (142.250.76.100) port 80 (#0)
> GET / HTTP/1.1
> Host: www.google.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 10 Mar 2021 02:37:46 GMT
< Expires: -1
< Cache-Control: private, max-age=0
< Content-Type: text/html; charset=ISO-8859-1
< P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< Server: gws
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< Set-Cookie: 1P_JAR=2021-03-10-02; expires=Fri, 09-Apr-2021 02:37:46 GMT; path=/; domain=.google.com; Secure
< Set-Cookie: NID=211=l3bDHkrWaIiQN03V4QI2bXrwHu0ZTMxoQgB1DDhIE2EfomQO6zrAPbu-5h6L6Ru60kQh0vtAog3iykbLvvtv28r8aVYiyMapXQWTexMArSIqRVQrcwDL4PnaoivdpE1_aL0rC6gohDpxFT-yQmk3jkE7EKtdSL_Dh7Z7nsnoqBM; expires=Thu, 09-Sep-2021 02:37:46 GMT; path=/; domain=.google.com; HttpOnly
< Accept-Ranges: none
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
<
{ [686 bytes data]
100 13716 0 13716 0 0 37889 0 --:--:-- --:--:-- --:--:-- 37889
* Connection #0 to host www.google.com left intact
* Closing connection 0

CURL multipart form post - HTTP error before end of send, stop sending

I am sending a multipart form POST via CURL from Linux shell script. The request works fine in Postman but from CURL it fails with:
Internal Server Error
then
HTTP error before end of send, stop sending
I even copy the Curl for Linux Shell code directly from Postman and paste into the shell script, so it should be exactly the same request that Postman is making.
Here is the command:
curl --request POST \
--no-alpn \
--url https://XXXXXXXXXXX/api/v1.0/XXXXX/XXXXXX/XXXXX \
--header 'accept: text/plain' \
--header 'cache-control: no-cache' \
--header 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
--header 'sessionid: $session_id' \
--form filename=XXXXXX.zip \
--form XXXXXX=XXXXXX \
--form file=#$file_path \
--trace-ascii /dev/stdout || exit $?
}
And here is the log from --trace-ascii:
https://XXXXXXXXXXXXXXXXX/api/v1.0/XXXXXX/XXXXX/XXXXXXXXX
Note: Unnecessary use of -X or --request, POST is already inferred.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0== Info: Trying XXX.XXX.XXX.XXXX...
== Info: Connected to XXXXXXXXXXXXXXXX.com (XX.XX.XX.XX) port 443 (#0)
== Info: found 148 certificates in /etc/ssl/certs/ca-certificates.crt
== Info: found 592 certificates in /etc/ssl/certs
== Info: SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
== Info: server certificate verification OK
== Info: server certificate status verification SKIPPED
== Info: common name: *.XXXXXX.com (matched)
== Info: server certificate expiration date OK
== Info: server certificate activation date OK
== Info: certificate public key: RSA
== Info: certificate version: #3
== Info: subject: OU=Domain Control Validated,CN=*.XXXXXX.com
== Info: start date: Mon, 15 Aug 2016 08:23:38 GMT
== Info: expire date: Thu, 15 Aug 2019 08:23:38 GMT
== Info: issuer: C=US,ST=Arizona,L=Scottsdale,O=GoDaddy.com\, Inc.,OU=http://certs.godaddy.com/repository/,CN=Go Daddy Secure Certificate Authority - G2
== Info: compression: NULL
=> Send header, 363 bytes (0x16b)
0000: POST /XXXXX/api/v1.0/XXXXXX/upload/XXXXXX HTTP/1.1
003b: Host: XXXXXX.XXXXXXX.com
0059: User-Agent: curl/7.47.0
0072: accept: text/plain
0086: cache-control: no-cache
009f: sessionid: $session_id
00b7: Content-Length: 1639
00cd: Expect: 100-continue
00e3: content-type: multipart/form-data; boundary=----WebKitFormBounda
0123: ry7MA4YWxkTrZu0gW; boundary=------------------------b059847fb557
0163: a899
0169:
<= Recv header, 23 bytes (0x17)
0000: HTTP/1.1 100 Continue
=> Send data, 387 bytes (0x183)
0000: --------------------------b059847fb557a899
002c: Content-Disposition: form-data; name="filename"
005d:
005f: xxxxxxxxxx.zip
006b: --------------------------b059847fb557a899
0097: Content-Disposition: form-data; name="XXXXXXXXXXX"
00cc:
00ce: XXXXXXXXXXXXXXXXXXXX
00ea: --------------------------b059847fb557a899
0116: Content-Disposition: form-data; name="file"; filename="XXXXXX.zip
0156: "
0159: Content-Type: application/octet-stream
0181:
=> Send data, 1204 bytes (0x4b4)
0000: PK........r~.K..D!....p.......output/XXXXXXX.XXXXX.....7Z..7Zux...
0040: ............{LSW....#.!`.9. F..Eh+.......JA..W.2.V...A.%>... #Q1
0080: T.....{Nb.]&..1.3M|.........w..z.]8..I.I>.....n?...\hM/.h..?oy^.
00c0: ..... ..:.>J..Q...N...*A...l`...."..N...#.P'........d..._.....L
0100: .].......z....N6.B......Y5t...Zd.V...}..l...........EC..$..e...W
0140: .V`.lV...p..d._.....S...............d`.l..}.....f[...{....`....M
0180: .....kN..[.4.2w.9.bN....q.8.'.K.......'..~........sI.....K...s.
01c0: ...U.'..d,.......>......T.5....|.$,)o'bIy{...pN.....K.o..[..cWp.
0200: c.#..B.S........d.I..P./.F..0....=4.......d..#{K$..#.^=.......
0240: *....Bi...i....8j!T......|.Ld...x....>......A...|.I.}>.....Yt=..
0280: ..Tp.q...O&.. .....Ac..V....a......f.G...!x.f.i.gu}.2i.4....NK..
02c0: .G;..k~......=*....g..c#..c.M.oW........-...vW.~#u...#....cz.bu=
0300: .."Bs.js\.z.1.....&|.MV..<a"4...IqRO.kKC.v.Gz.....].G.\.|...:om
0340: .C.v5G..X].kw..\....R/.........C.X].5<.B.\'....z.O|#.v.P\......
0380: ^...f~........9....YG~fum}....^,K.......F.vmIl....hI."h.FM.....f
03c0: ....Z...`um.}E...1;......_....yF.xV...BDh...U..z...*.o.`O..V.W.6
0400: ..kf.n...*.{..].].c~.w~K......4I.k.Y.....r.wV.................F
0440: .v..O..OPK..........r~.K..D!....p.....................output/xxxxxxx
.mldUT.....7Zux.............PK..........V...H.....
=> Send data, 48 bytes (0x30)
0000:
0002: --------------------------b059847fb557a899--
<= Recv header, 36 bytes (0x24)
0000: HTTP/1.1 500 Internal Server Error
<= Recv header, 15 bytes (0xf)
0000: Server: nginx
<= Recv header, 37 bytes (0x25)
0000: Date: Mon, 18 Dec 2017 15:15:56 GMT
<= Recv header, 26 bytes (0x1a)
0000: Content-Type: text/plain
<= Recv header, 28 bytes (0x1c)
0000: Transfer-Encoding: chunked
<= Recv header, 24 bytes (0x18)
0000: Connection: keep-alive
100 1639 0 0 100 1639 0 2269 --:--:-- --:--:-- --:--:-- 2266<= Recv header, 29 bytes (0x1d)
0000: X-FRAME-OPTIONS: SAMEORIGIN
<= Recv header, 83 bytes (0x53)
0000: Set-Cookie: JSESSIONID=XXXXXXXXXXXXXXXXXXXXXXXX; Path=/;
0040: Secure; HttpOnly
== Info: HTTP error before end of send, stop sending
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 106 bytes (0x6a)
0000: 64
0004: <ErrorResponse><key/><localizedMessage/><httpError>Internal Serv
0044: er Error</httpError></ErrorResponse>
<= Recv data, 5 bytes (0x5)
0000: 0
0003:
I should add that the CURL command is being run from a Docker container.
There's a problem in you curl command : $session_id have single quotes around, so the variable will never be evaluated.
"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[#]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words
In my particular case the problem was that I was sending session ID surrounded in double quotes, so the server failed to parse as a number, threw an exception and rejected the request. Had to get hold of the server logs to figure that out.
Reason session ID was in double quotes was because earlier on in the code, I was setting the session ID using:
someJson | jq '.sessionId'
If you do this, jq will return the result in double quotes. To get the value without the double quotes, use:
someJson | jq -r '.sessionId'

Varnish Breaking Social Sharing

Facebook uses a curl with range option to retrieve HTML of a page for sharing. Varnish is only returning page header info and not the html. This is the result I would say 75% to 80% of the time. Every once in a while it returns the correct Result
Anyone have an Idea how to fix this.
Example
#curl -v -H Range:bytes=0-524288 http://americanactionnews.com/articles/huge-protest-calls-for-death-to-usa-demands-isis-control
* Trying 52.45.101.42...
* TCP_NODELAY set
* Connected to americanactionnews.com (52.45.101.42) port 80 (#0)
> GET /articles/huge-protest-calls-for-death-to-usa-demands-isis-control HTTP/1.1
> Host: americanactionnews.com
> User-Agent: curl/7.51.0
> Accept: */*
> Range:bytes=0-524288
>
< HTTP/1.1 206 Partial Content
< Content-Type: text/html; charset=utf-8
< Status: 200 OK
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Date: Fri, 23 Dec 2016 16:39:46 GMT
< Cache-Control: max-age=300, public
< X-Request-Id: 8007fb76-3878-430d-845c-06a8710ae1ae
< X-Runtime: 0.247333
< X-Powered-By: Phusion Passenger 4.0.55
< Server: nginx/1.6.2 + Phusion Passenger 4.0.55
< X-Varnish-TTL: 300.000
< X-Varnish: 65578
< Age: 0
< Via: 1.1 varnish-v4
< X-Cache: MISS
< Transfer-Encoding: chunked
< Connection: keep-alive
< Accept-Ranges: bytes
< Content-Range: bytes 0-15/16
<
* Curl_http_done: called premature == 0
* Connection #0 to host americanactionnews.com left intact
We have found the answer and made some modifications to our varnish based on the following link and it seems to be working.
https://info.varnish-software.com/blog/caching-partial-objects-varnish
It looks like it is actually your backend server (Nginx) that is the problem. Especially considering your mentioned hit-rate-like success rate :) Plus, the failure's example is a MISS (delivered from backend server).
Make sure that you don't have anything in your Nginx configuration that prevents range requests, i.e. one popular thing that breaks it is:
ssi on;
ssi_types *;

Varnish error : no backend connection

I tried to install Varnish on my debian 7 with apache2.
But when i type www.mydomain.com:6081 to test the connection, i got a 503 error service unavaible.
Varnish log says :
12 Hash c www.mywebsite.com:6081
12 VCL_return c hash
12 VCL_call c pass pass
12 FetchError c no backend connection
12 VCL_call c error deliver
12 VCL_call c deliver deliver
12 TxProtocol c HTTP/1.1
12 TxStatus c 503
My etc/varnish/default.vcl file :
(only one backend for now)
backend site1 {
.host = "92.243.5.12"; // ip adress for www.mydomain.com
.port = "8080";
.connect_timeout = 6000s;
.first_byte_timeout = 6000s;
.between_bytes_timeout = 6000s;
}
# Default backend is set to site1
set req.backend = site1;
My etc/default/varnish file :
DAEMON_OPTS="-a :80
-T localhost:6082
-f /etc/varnish/default.vcl
-S /etc/varnish/secret
-p thread_pool_add_delay=2
-p thread_pools=4
-p thread_pool_min=200
-p thread_pool_max=4000
-p cli_timeout=25
-p session_linger=100
-s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"
Thank you very much
In your conf file, you are making varnish listen to port 80 and you are sending requests to port 6081, that may be a reason.

Resources