Curl Command in Shell Script in Azure Pipeline not working - azure

Question

I'm trying to run a script which provides the status code of some url, using azure pipeline.
My Azure Yaml file:
pool:
vmImage: ubuntu-16.04
steps:
- script: echo Hello
displayName: ' Welcome'
- script: cat webpages.txt
displayName: 'display file'
- script: curl -s -w '%{http_code}\n' -o /dev/null https://www.google.co.in
displayName: 'Checking Curl Code'
- script: cat -v script.sh
displayName: 'Cariage retrun'
- task: ShellScript#2
inputs:
scriptPath: script.sh
My Script.sh file
#!/bin/sh
while read line ; do echo "$line - `curl -s -w '%{http_code}\n' -o /dev/null $line`" ;done < webpages.txt
webpages.txt file
https://www.vodafone.co.uk/good-stuff
https://www.vodafone.co.uk/help-and-information/cancel-your-account
https://www.vodafone.co.uk/help-and-information/complaints
https://www.vodafone.co.uk/help-and-information/complaints/code-of-practice
https://www.vodafone.co.uk/help-and-information/costs-and-charges
https://www.vodafone.co.uk/help-and-information/costs-and-charges/call-and-text-charges
https://www.vodafone.co.uk/help-and-information/costs-and-charges/data-charges
Problem
When I run my pipeline, the curl command is not working
and output comes as
2020-11-17T09:59:05.4094324Z ##[section]Starting: ShellScript
2020-11-17T09:59:05.4102534Z ==============================================================================
2020-11-17T09:59:05.4102904Z Task : Shell script
2020-11-17T09:59:05.4103204Z Description : Run a shell script using Bash
2020-11-17T09:59:05.4103453Z Version : 2.165.2
2020-11-17T09:59:05.4103701Z Author : Microsoft Corporation
2020-11-17T09:59:05.4104086Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/utility/shell-script
2020-11-17T09:59:05.4104506Z ==============================================================================
2020-11-17T09:59:05.7637654Z [command]/bin/bash /home/vsts/work/1/s/script.sh
2020-11-17T09:59:05.7692023Z https://www.vodafone.co.uk/good-stuff
2020-11-17T09:59:05.7700780Z - 000
2020-11-17T09:59:05.7797276Z https://www.vodafone.co.uk/help-and-information/cancel-your-account
2020-11-17T09:59:05.7798378Z - 000
2020-11-17T09:59:05.7851183Z https://www.vodafone.co.uk/help-and-information/complaints
2020-11-17T09:59:05.7866944Z - 000
2020-11-17T09:59:05.7908420Z https://www.vodafone.co.uk/help-and-information/complaints/code-of-practice
2020-11-17T09:59:05.7909144Z - 000
2020-11-17T09:59:05.7967261Z https://www.vodafone.co.uk/help-and-information/costs-and-charges
2020-11-17T09:59:05.7967920Z - 000
2020-11-17T09:59:05.8023329Z https://www.vodafone.co.uk/help-and-information/costs-and-charges/call-and-text-charges
2020-11-17T09:59:05.8024443Z - 000
2020-11-17T09:59:05.8095527Z https://www.vodafone.co.uk/help-and-information/costs-and-charges/data-charges
but if I replace my curl with
curl -s -w '%{http_code}\n' -o /dev/null https://www.vodafone.co.uk/good-stuff
it gives the output at 200.

Answer 1

When I'm running this locally it works perfectly:
$ while read line ; do echo "$line - `curl -s -w '%{http_code}\n' -o /dev/null $line`" ;done < webpages.txt
https://www.vodafone.co.uk/good-stuff - 200
https://www.vodafone.co.uk/help-and-information/cancel-your-account - 200
(...)
I notice that a \n character is printed in your echo $line, that's probably causing the issue. What could solve it, is:
replacing $line by ${line},
replacing echo by echo -n to omit newlines.

Answer 2

So this is what I did,
I replaced the whole curl command with a simple curl $line
It gave me an error : curl: (3) Illegal characters found in URL
So, I figured my URL is adding some unrequired values.
I replaced my do with
line=${line%$'\r'}
while read line ; do line=${line%$'\r'} ; echo "$line - `curl -s -w '%{http_code}\n' -o /dev/null $line`"; done < webpages.txt
and Voila it works!