Can anyone help me in figuring out how can I extract body from the headers section in the following response using bash scripting.
I have tried awk, sed, grep ... by looking some of the existing solutions here on SO, but with little success.
Let me know if I need to provide any additional info.
HTTP/1.1 200 OK
Cache-Control: max-age=604800
Content-Type: text/html
Date: Mon, 24 Jul 2017 10:16:19 GMT
Etag: "359670651+gzip+ident"
Expires: Mon, 31 Jul 2017 10:16:19 GMT
Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT
Server: ECS (iad/182A)
Vary: Accept-Encoding
X-Cache: HIT
Content-Length: 1270
<!doctype html>
<html>
<head>
<title>Example Domain</title>
<meta charset="utf-8" />
<meta http-equiv="Content-type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<style type="text/css">
body {
background-color: #f0f0f2;
margin: 0;
padding: 0;
font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
}
div {
width: 600px;
margin: 5em auto;
padding: 50px;
background-color: #fff;
border-radius: 1em;
}
a:link, a:visited {
color: #38488f;
text-decoration: none;
}
#media (max-width: 700px) {
body {
background-color: #fff;
}
div {
width: auto;
margin: 0 auto;
border-radius: 0;
padding: 1em;
}
}
</style>
</head>
<body>
<div>
<h1>Example Domain</h1>
<p>This domain is established to be used for illustrative examples in documents. You may use this
domain in examples without prior coordination or asking for permission.</p>
<p>More information...</p>
</div>
</body>
</html>
I didn't explain correctly, what I meant is to extract the body from http response and not body of the html response. In other words, how can extract http's body response in general (the part in the response after \r\n\r\n), this code is just for demo purposes...
sed -n '/<body>/,/<\/body>/p' filename
Print everything from to
To output the body inner HTML(without the body tag):
sed -n '/<body/,/<\/body>/{//!p}' file
Related
It doesn't give error but it doesn't work. I cannot see the file on the other server.
What should I write to upload file with curl with no user interaction?
$curl -k -F 'myfile=#/opt/test.txt' -H 'HTTP_AUTH_LOGIN:<destinationHostUsername>' -H 'HTTP_AUTH_PASSWD:<destinationHostPassword>' <destinationHostName>/
This html code below is the output when I run that curl command. I tried many lines but I couldn't do it. Can you guys can help me out?
(says "It looks like your post is mostly code; please add some more details." So I'm gonna write lots of things you don't have to read it
It looks like your post is mostly code; please add some more details.)
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<title>Test Page for the Apache HTTP Server on Red Hat Enterprise Linux</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<style type="text/css">
/* <![CDATA[ */
body {
background-color: #fff;
color: #000;
font-size: 0.9em;
font-family: sans-serif, helvetica;
margin: 0;
padding: 0;
}
:link {
color: #c00;
}
:visited {
color: #c00;
}
a:hover {
color: #f50;
}
h1 {
text-align: center;
margin: 0;
padding: 0.6em 2em 0.4em;
background-color: #900;
color: #fff;
font-weight: normal;
font-size: 1.75em;
border-bottom: 2px solid #000;
}
h1 strong {
font-weight: bold;
}
h2 {
font-size: 1.1em;
font-weight: bold;
}
hr {
display: none;
}
.content {
padding: 1em 5em;
}
.content-columns {
/* Setting relative positioning allows for absolute positioning for sub-classes */
position: relative;
padding-top: 1em;
}
.content-column-left {
/* Value for IE/Win; will be overwritten for other browsers */
width: 47%;
padding-right: 3%;
float: left;
padding-bottom: 2em;
}
.content-column-left hr {
display: none;
}
.content-column-right {
/* Values for IE/Win; will be overwritten for other browsers */
width: 47%;
padding-left: 3%;
float: left;
padding-bottom: 2em;
}
.content-columns>.content-column-left,
.content-columns>.content-column-right {
/* Non-IE/Win */
}
img {
border: 2px solid #fff;
padding: 2px;
margin: 2px;
}
a:hover img {
border: 2px solid #f50;
}
/*]]>*/
</style>
</head>
<body>
<h1>Red Hat Enterprise Linux <strong>Test Page</strong>
</h1>
<div class="content">
<div class="content-middle">
<p>This page is used to test the proper operation of the Apache HTTP server after it has been installed. If you can read this page, it means that the Apache HTTP server installed at this site is working properly.</p>
</div>
<hr />
<div class="content-columns">
<div class="content-column-left">
<h2>If you are a member of the general public:</h2>
<p>The fact that you are seeing this page indicates that the website you just visited is either experiencing problems, or is undergoing routine maintenance.</p>
<p>If you would like to let the administrators of this website know that you've seen this page instead of the page you expected, you should send them e-mail. In general, mail sent to the name "webmaster" and directed to the website's domain should reach the appropriate person.</p>
<p>For example, if you experienced problems while visiting www.example.com, you should send e-mail to "webmaster#example.com".</p>
<p>For information on Red Hat Enterprise Linux, please visit the Red Hat, Inc. website. The documentation for Red Hat Enterprise Linux is available on the Red Hat, Inc. website. </p>
<hr />
</div>
<div class="content-column-right">
<h2>If you are the website administrator:</h2>
<p>You may now add content to the directory <tt>/var/www/html/</tt>. Note that until you do so, people visiting your website will see this page, and not your content. To prevent this page from ever being used, follow the instructions in the file <tt>/etc/httpd/conf.d/welcome.conf</tt>. </p>
<p>You are free to use the image below on web sites powered by the Apache HTTP Server:</p>
<p align="center">
<a href="http://httpd.apache.org/">
<img src="/icons/apache_pb2.gif" alt="[ Powered by Apache ]" />
</a>
</p>
</div>
</div>
</div>
</body>
</html>
I have a "testing.txt" file that has this following string
example.com
i want to curl that url by using this command
curl $testing.txt
and i expect to get this output
<!doctype html>
<html>
<head>
<title>Example Domain</title>
<meta charset="utf-8" />
<meta http-equiv="Content-type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<style type="text/css">
body {
background-color: #f0f0f2;
margin: 0;
padding: 0;
font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
}
div {
width: 600px;
margin: 5em auto;
padding: 2em;
background-color: #fdfdff;
border-radius: 0.5em;
box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);
}
a:link, a:visited {
color: #38488f;
text-decoration: none;
}
#media (max-width: 700px) {
div {
margin: 0 auto;
width: auto;
}
}
</style>
</head>
<body>
<div>
<h1>Example Domain</h1>
<p>This domain is for use in illustrative examples in documents. You may use this
domain in literature without prior coordination or asking for permission.</p>
<p>More information...</p>
</div>
</body>
</html>
so the question is, how can i curl a URL by using a string that in a file?
Basically,
To read content file:
cat the_file_to_read
So,
curl --url "$(cat the_file_to_read)" --output "the_file_to_write"
Of course, the input file content must be contains the complete URL... or not. You could add the protocol in cURL command:
curl --url "https://$(cat the_file_to_read)" --output "the_file_to_write"
To prevent more than one line in input file, you need one or mode filters like that:
curl --url "https://$(cat the_file_to_read | head -1)" --output "the_file_to_write"
You also can read HTTP response code like this:
HTTP_RESPONSE_CODE=$(curl --url "https://$(cat the_file_to_read | head -1)" --output "the_file_to_write" --write-out '%{response_code}')
echo "HTTP_RESPONSE_CODE=${HTTP_RESPONSE_CODE}"
See man curl for more explanations, more options...
So a git issue had me roll back about two weeks of work -
Im currently trying to pass an array of about 3300 string to a handlebars template then trying to print that as a pdf - my issue is I'm pretty sure my pupepteer URL is being cut off at 3000 characters. Im at a loss for a workaround.
<<<< my data logs as -----> Array(3330) [Object, Object, Object, Object, Object, Object, Object, Object, …] >>>
var templateHtml = fs.readFileSync(path.join(process.cwd(), 'template.html'), 'utf8');
var template = handlebars.compile(templateHtml);
var html = template(data);
await page.goto(`data:text/html;charset=UTF-8,${html}`, {
waitFor:10000
});
before my git meltdown I was printing out 90 page PDF's and I just cant figure out what I did before.
Answer
Your problem seems to be the Data URI length limitation. It is 2MB in case of Chromium. So if your html exceeds the limit it will be trimmed or even not rendered at all.
I suggest to use page.setContent with the same content as it has no upper limit.
Example
Note: setContent needs a string as input, I've just copied the source of example.com.
const puppeteer = require('puppeteer')
async function fn() {
const browser = await puppeteer.launch({ headless: true })
const page = await browser.newPage()
await page.goto('data:text/html,<h1>Template</h1>')
await page.waitFor(2000)
await page.setContent(
'<!doctype html> <html> <head> <title>Example Domain</title> <meta charset="utf-8" /> <meta http-equiv="Content-type" content="text/html; charset=utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <style type="text/css"> body { background-color: #f0f0f2; margin: 0; padding: 0; font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif; } div { width: 600px; margin: 5em auto; padding: 2em; background-color: #fdfdff; border-radius: 0.5em; box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02); } a:link, a:visited { color: #38488f; text-decoration: none; } #media (max-width: 700px) { div { margin: 0 auto; width: auto; } } </style> </head> <body> <div> <h1>Example Domain</h1> <p>This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.</p> <p>More information...</p> </div> </body> </html>'
)
await page.pdf({ path: 'page.pdf' })
await browser.close()
}
fn()
I have a website that is primarily html5 and css3, and the pages look good when I run them locally from my pc on chrome and internet explorer 11.0.47, but when I put it on the Linux server and run the page from there on my own pc, the page loses the css in IE but is fine in chrome.
I don't have a lot of experience with html5/css3. What's going on here? I tried googling it, but don't see any online help.
The web page is start of web site. As you can see, in IE, Map Home isn't shown as a button in nav to the left, and nav lost it's brown background. It also lost the header background. If you look at the link in chrome, it has those.
Any ideas? Could there be a directional issue with use of the media folder when IE pulls it from Linux but not chrome?
**Added:
I see this on the server log, but when I list the location on the server, the file exists there. The css file (catvStyles.css) referring to the globe file is in the same dir as index.html, and the media dir is in the dir of index.html.
File does not exist:
/opt/apps/html/catv/media/globe_transp_gradation.png), url(..,
referer: http://ltrkarkvm391.mgmt.windstream.net/catv/index.html
**
This looks similar to other web page if you can't reach the link. The problem is my linux hosted site isn't showing the brown nav to the left, it's not showing the metal header globe, and windmill pictures, and it's not showing Map Home in a button (it's as a link without the button), and it's not showing the picture in the right side to the right (it's in the body now).
This is the index.html page:
<!DOCTYPE html>
<html lang="en">
<head>
<link type="text/css" rel="stylesheet" href="FunStuff/catvStyles.css">
<title>CATV Monitoring</title>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="Wi">
<meta name="keywords" content="catv, cmts, snmp, modem, dhcp, rf, status, map">
<meta name="author" content="Wi, Michele, Adam">
<style type="text/css">
</style>
<!--[if lt IE 9]>
<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"> </script>
<![endif]-->
</head>
<body>
<div id="wrapper">
<header role="banner" class="bgFun">
<h1>Wi</h1>
<h2 id="headerH2Pos">CATV Monitoring System</h2>
</header>
<div id="page">
<nav role="navigation" id="navBakImg">
<ul>
<li>Map Home</li> <!--MapHome.html-->
</ul>
</nav>
<main role="main" id="middle">
<h2>Purpose</h2>
<p>To provide CATV Monitoring to our internal customers</p>
</p>
<h2>Mapping </h2>
<p>Click the "Map Home" link on the left to see how the network is doing </p>
<img class="imgFix" src="media/WIN_Vert_Green_Logo.png" height:"18" width:"15" alt="Logo" title="Wi" >
</p>
</main>
<aside role="complementary">
<img class="imgFix" src="media/WIN_Vert_Green_Logo.png" height:"18" width:"15" alt="Logo" title="Wi" >
<h2>How To Start</h2>
<p>See directions</p>
</aside>
</div> <!-- end of flex container -->
<footer role="contentinfo">
<img class="tree" src="media/Tree_Branches_And_Roots_clip_art_small.png" height="50" width="50" alt="Tree" title="Created by Michele " >
<br>Copyright © 2017
<script>
document.write('Last Modified: ' + document.lastModified);
</script>
<br>
</footer>
</div>
</body>
</html>
This is the css catvStyles.css:
body {font-family: Verdana, Arial, sans-serif;
background-color: #330000;
background-image: url(../media/green.gif);
}
#middle{}
header, h1, wrapper { margin-top: 0; }
#wrapper { background-color: #F4E8BC;
/*background-color: #00ffff;*/
width: 90%;
margin: auto;
color: #003300;/*color: #330000;*/
}
h1, h2 { color: #003300; }
header {
background-repeat: no-repeat;
background-color:transparent;
}
.bgFun{background-image: url(../media/globe_transp_gradation.png),
url(../media/windmill_transp_gradation.png),url(../media/MetalGalvanized0014_M.jpg);
background-position: left, right;
/*width: 80%;
height:80%;*/
background-size:45%,30%,cover;
background-repeat:no-repeat;
background-color:transparent;}
h1 { text-align: center;
font-size: 300%;
padding: 5% 0;
text-shadow: 3px 3px 3px #F4E8BC;
}
nav, main, aside { padding: 0 1em;
}
.imgFix { width: 25%;
height: auto;
}
nav ul { list-style-type: none;
font-size: 1.2em;
padding-left: 0;
}
/*nav a { text-decoration: none;}*/
nav a {
text-decoration: none;
background-color: #666666;
display:block;
text-align:center;
width:100%;
margin:1em auto;
border:solid .08em #339900
}
nav a:link { color: #330000; }
nav a:visited { color: #003300; }
nav a:hover { color: #996600;
background-color:#000000;}
footer { text-align: center;
font-size: 80%;
font-style: italic;
color: #003300;
padding: 2.5%;
}
#page{display: -webkit-flex;
display: flex;
-webkit-flex-direction: row;
flex-direction: row;
}
#navBakImg{border: .1em solid #000000;
padding-right:1em;
background-image: url(../media/CardboardPlain0016_2_M.jpg);
font-size:90%;}
nav{-webkit-flex: 1;
flex: 1;
}
main{-webkit-flex: 7;
flex:7;
}
aside{-webkit-flex: 2;
flex: 2;
}
.tree { width:3em;
height: auto;
float:right;}
#media only screen and (max-width: 1024px) {
body { margin: 0; padding: 0; }
#wrapper { width: auto; margin: 0; }
h1 { font-size: 200%; }
nav li { padding: 0 0.5em;
font-size: 1.2em;
}
#page{-webkit-flex-direction: column;
flex-direction: column;
}
nav ul{display: -webkit-flex;
display: flex;
-webkit-flex-direction: row;
flex-direction: row;
webkit-justify-content: center;
justify-content: center;
}
.imgFix { width: 30%;
height: auto;}
}
#media only screen and (max-width: 768px) {
nav{-webkit-order: 1;
order: 1;
}
nav ul{-webkit-flex-wrap: wrap;
flex-wrap: wrap;
}
section{display:none;}
.imgFix { width: 35%;
height: auto;}
}
header, main, nav, footer, figure, figcaption { display: block; }
#headerH2Pos{position:relative;
left:26%;
}
EDIT: Looks like it's something else, as Michele says it works in Chrome but not IE when served from Linux.
Judging by your filenames, you may have just encountered a Windows => Linux gotcha:
The Linux filesystem is Case Sensitive, Windows is case insensitive.
You'll need to double check that you're using the right casing in your URLs, else the file won't be found when Linux goes to look for it.
I had to add this line and it was fixed:
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
Hi I'm new to nodejs so I using the npm library instagram-node-lib... I was using this for long time and tested it works correctly before but I receive an error from the terminal.
in _request
SyntaxError: Unexpected token < occurred: <!DOCTYPE html>
<html lang="en" id="facebook">
<head>
<title>Facebook | Error</title>
<meta charset="utf-8">
<meta http-equiv="cache-control" content="no-cache">
<meta http-equiv="cache-control" content="no-store">
<meta http-equiv="cache-control" content="max-age=0">
<meta http-equiv="expires" content="-1">
<meta http-equiv="pragma" content="no-cache">
<meta name="robots" content="noindex,nofollow">
<style>
html, body {
color: #141823;
background-color: #e9eaed;
font-family: Helvetica, Lucida Grande, Arial,
Tahoma, Verdana, sans-serif;
margin: 0;
padding: 0;
text-align: center;
}
#header {
background: #3b5998
url('//static.xx.fbcdn.net/images/logo-fb-trans-92x20.png')
center 11px no-repeat;
border-bottom: 1px solid #133783;
display: block;
height: 38px;
position: relative;
}
h1 {
font-size: 18px;
}
p {
font-size: 13px;
}
#footer {
border-top: 1px solid #ddd;
color: #9197a3;
font-size: 12px;
padding: 5px 8px 6px 0;
}
</style>
</head>
<body>
<div>
<a id="header" href="//www.facebook.com/"></a>
</div>
<div id="core">
<h1 id="sorry">Sorry, something went wrong.</h1>
<p id="promise">
We're working on it and we'll get it fixed as soon as we can.
</p>
<p id="back-link">
<a id="back" href="//www.facebook.com/">Go Back</a>
</p>
<div id="footer">
Facebook
<span id="copyright">
© 2014
</span>
<span id="help-link">
·
<a id="help" href="//www.facebook.com/help/">Help Center</a>
</span>
</div>
</div>
<script>
document.getElementById('back').onclick = function() {
if (history.length > 1) {
history.back();
return false;
}
};
// Adjust the display based on the window size
if (window.innerHeight < 80 || window.innerWidth < 80) {
// Blank if window is too small
document.body.style.display = 'none';
};
if (window.innerWidth < 200 || window.innerHeight < 150) {
document.getElementById('header').style.background = '#3b5998';
document.getElementById('header').style.height = '8px';
document.getElementById('back-link').style.display = 'none';
document.getElementById('help-link').style.display = 'none';
};
if (window.innerWidth < 200) {
document.getElementById('sorry').style.fontSize = '16px';
};
if (window.innerWidth < 150) {
document.getElementById('promise').style.display = 'none';
};
if (window.innerHeight < 150) {
document.getElementById('sorry').style.margin = '4px 0 0 0';
document.getElementById('sorry').style.fontSize = '14px';
document.getElementById('promise').style.display = 'none';
};
</script>
</body>
</html>
is there a way to rectify this error shown from the terminal?