Click here to view table
I think this is a simple task, but I'm a biologist who only knows a teeny bit of code and after several days of trying to figure this out, I'm out of ideas.
Using terminal on a Mac. I have a CSV file that I want to split into separate files by row (162 rows) and I want to name the file by the content of the first and second column (genus_species). Then I need all 162 genus_species to be saved as HTML files.
I have only attempted the "splitting" part with Ruby (recommendation from StackExchange/overflow). Below are some of my attempts. They are frankensteins of helpful-ish forums, and after each I made a little comment on why it did not work.
Example HTML
<!DOCTYPE html>
<html><head>
<meta charset="UTF-8">
<script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script></head>
<body>
<h1><em><!-- Species name --></em> - <!-- Common name --></h1>
<h2>Status</h2>
<p></p>
<h2>Info</h2>
<p></p>
<h2>Time of year this bee is seen</h2>
<p></p>
<h2>Identification</h2>
<p></p>
<h3>Similar Species</h3>
<p></p>
<h2>Flowers</h2>
<p></p>
<h2>Sociality</h2>
<p></p>
<h2>Nest</h2>
<p></p>
<div id="refs" class="references">
--<br>More information:<br> <!-- Bug Guide --></div>
</body></html>
More Info Based on Comments
Here are some lines copied from the text file:
Genus,species,Common name,Status,Info,Time of year this bee is seen,Identification,Similar Species,Flowers,Sociality,Nest,Bug Guide,Discover Life,Other,
Agapostemon,melliventris,Honey-tailed Striped-Sweat bee,Secure G5,Excavates into deep burrows in ground nests,March-December,Agapostemon males have black and yellow stripes on the abdomen. Females have a yellow band on the lower margin of the clypeus.,All other Agapostemon species,Wide variety of plants,Solitary,"Deep, underground excavation",https://bugguide.net/node/view/70932,https://www.discoverlife.org/20/q?search=Agapostemon+melliventris,https://explorer.natureserve.org/Taxon/ELEMENT_GLOBAL.2.928401/Agapostemon_melliventris,
Agapostemon,sericeus,Silky Striped Sweat Bee,Secure G5,"Not choosy about lawn, as long as flowers are present",April-October,Agapostemon males have black and yellow stripes on the abdomen. A. sericeus males have a tooth on its hind femur. Female has metallic green abdomen.,All other Agapostemon species,Wide variety of plants,Solitary,Ground-nester in loamy soils,https://bugguide.net/node/view/83023,https://www.discoverlife.org/mp/20q?search=Agapostemon+sericeus,https://www.sharpeatmanguides.com/sweat-bees,
Agapostemon,splendens,Brown-winged Striped-Sweat Bee,Secure G5,This is the most common Agapostemon found in the southeast region,April-October,Agapostemon males have black and yellow stripes on the abdomen. A. splendens have brown wings. The female abdomen is often somewhat bluish.,All other Agapostemon species,"Jacquemontia reclinata, wide variety of plants",Solitary,Ground-nester in sandy soils,https://bugguide.net/node/view/74478,https://www.discoverlife.org/mp/20q?search=Agapostemon+splendens,,
Updated code I've tried based on comments.
This worked and I think it's heading in the direction I want, but it's hard to tell in the terminal window:
f = File.new("bee_key_fact_sheet .csv")
f.each_line { |line| puts line }
Currently playing with some kind of File.write line to add here and then close?
Attempt #1
file = File.open("bee_key_fact_sheet.csv")
awk
'(NR==1){header=$0;next}
(NR%l==2) {
close(file);
file=sprintf("%s.%0.5d.csv",FILENAME,++c)
sub(/csv[.]/,"",file)
print header > file
}
{f.write}'
File.close
#AWK not recognized, asks to "display all possibilities (y/n)" I tried returning "y" and "yes" and both times it says my answer is not recognized
Attempt #2
file_data = File.read("bee_key_fact_sheet.csv").split
#This works but splits by each comma
Attempt #3
file_data = File.foreach("bee_key_fact_sheet.csv") { |line| puts line}.split
#This returned something slightly less messy than splitting by each comma but got this error message "undefined method `split' for nil:NilClass"
Attempt #4
bee_key_fact_sheet.csv.foreach('so1.csv', :headers => true, :col_sep => ",", :skip_blanks => true) do |row|
id, name = row[0], row[1]
unless (id =~ /#/)
names = name.split
end
#This returned nothing
Your example of CSV input (bee_key_fact_sheet.csv):
Genus,species,Common name,Status,Info,Time of year this bee is seen,Identification,Similar Species,Flowers,Sociality,Nest,Bug Guide,Discover Life,Other,
Agapostemon,melliventris,Honey-tailed Striped-Sweat bee,Secure G5,Excavates into deep burrows in ground nests,March-December,Agapostemon males have black and yellow stripes on the abdomen. Females have a yellow band on the lower margin of the clypeus.,All other Agapostemon species,Wide variety of plants,Solitary,"Deep, underground excavation",https://bugguide.net/node/view/70932,https://www.discoverlife.org/20/q?search=Agapostemon+melliventris,https://explorer.natureserve.org/Taxon/ELEMENT_GLOBAL.2.928401/Agapostemon_melliventris,
Agapostemon,sericeus,Silky Striped Sweat Bee,Secure G5,"Not choosy about lawn, as long as flowers are present",April-October,Agapostemon males have black and yellow stripes on the abdomen. A. sericeus males have a tooth on its hind femur. Female has metallic green abdomen.,All other Agapostemon species,Wide variety of plants,Solitary,Ground-nester in loamy soils,https://bugguide.net/node/view/83023,https://www.discoverlife.org/mp/20q?search=Agapostemon+sericeus,https://www.sharpeatmanguides.com/sweat-bees,
Agapostemon,splendens,Brown-winged Striped-Sweat Bee,Secure G5,This is the most common Agapostemon found in the southeast region,April-October,Agapostemon males have black and yellow stripes on the abdomen. A. splendens have brown wings. The female abdomen is often somewhat bluish.,All other Agapostemon species,"Jacquemontia reclinata, wide variety of plants",Solitary,Ground-nester in sandy soils,https://bugguide.net/node/view/74478,https://www.discoverlife.org/mp/20q?search=Agapostemon+splendens,,
In this CSV, all the lines (including the header) end with a comma, so the last column probably doesn't mean anything and is to be discarded.
Also, you have commas inside the data (fields with double-quotes), so you'll need a real CSV parser to read the content of the file. BTW, you're right in choosing Ruby for this task because it includes a CSV parser in its core library.
Here's one way of reading your CSV (Edit: fixed CSV#Row conversion for older Rubys):
require 'csv'
filepath = 'bee_key_fact_sheet.csv'
CSV.foreach(filepath, headers: true) do |row|
genus, species = row[0], row[1]
#data = row[0...-1] # NOTE: not sure about the Ruby version compatibility
data = row.to_hash.values[0...-1]
filename = "#{genus}_#{species}.txt".tr("\0/",'')
filecontent = " * #{data.join("\n * ")}"
puts "\n#{filename}:\n#{filecontent}"
end
About tr("\0/",''): The characters that are allowed in a filename depend on the filesystem. All the filesystems (that I know of) ban at least the NULL-byte and the slash characters, so I strip them (but you may want to strip a few more).
Question: What exactly is the expected HTML output? A table row?
Update: HTML generation
When generating content programmatically, it's fundamental to escape your data for the right format/language/context. In Ruby you can escape HTML with CGI.escapeHTML
Your example of HTML output:
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
</head>
<body>
<h1><em><!-- Species name --></em> - <!-- Common name --></h1>
<h2>Status</h2>
<p></p>
<h2>Info</h2>
<p></p>
<h2>Time of year this bee is seen</h2>
<p></p>
<h2>Identification</h2>
<p></p>
<h3>Similar Species</h3>
<p></p>
<h2>Flowers</h2>
<p></p>
<h2>Sociality</h2>
<p></p>
<h2>Nest</h2>
<p></p>
<div id="refs" class="references">
--
<br>More information:
<br> <!-- Bug Guide -->
</div>
</body>
</html>
I'll make a few changes to the HTML:
Add a title to the page.
Remove MathJax which seams unnecessary.
Convert the <h3> tag to <h2> because you use it only for "Similar Species". Changing it also permits the use of a loop while generating the HTML.
You have 2 links in the CSV that you don't use in the HTML: "Discover Life" and "Other", don't you want to display them ? I added the code for that ;-)
OK, first, you create a function that, given a CSV row, generates the corresponding HTML. Here I use ERB templating but you can do it directly with string literals (Edit: fixed ERB#result arguments for Ruby < 2.4.0):
require 'cgi'
require 'erb'
def renderHTML row
htmlsafe = row.each_with_object({}) { |(k,v),h| h[k] = CGI.escapeHTML v if v }
template = <<-'EOF'
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title><%= "#{htmlsafe['Genus']} #{htmlsafe['species']}" %></title>
</head>
<body>
<h1><em><%= "#{htmlsafe['Genus']} #{htmlsafe['species']}" %></em> - <%= htmlsafe['Common name'] %></h1>
<% for key in ['Status','Info','Time of year this bee is seen','Identification','Similar Species','Flowers','Sociality','Nest'] %>
<h2><%= key %></h2>
<p><%= htmlsafe[key] %></p>
<% end %>
<div id="refs" class="references">
--
<br>More information:
<% for key in ['Bug Guide', 'Discover Life', 'Other'].select{ |k| htmlsafe[k] } %>
<br><%= key %>
<% end %>
</div>
</body>
</html>
EOF
#ERB.new(template, trim_mode: "<>").result(binding) # NOTE: only for Ruby >= 2.4.0
ERB.new(template, nil, "<>").result(binding)
end
Then you can call the previous function while reading each row of your CSV file:
require 'csv'
filepath = 'bee_key_fact_sheet.csv'
CSV.foreach(filepath, headers: true) do |row|
filename = "#{row['Genus']}_#{row['species']}.html".tr("\0/",'')
html = renderHTML row
puts "\n# #{filename}\n#{html}"
#File.write(filename, html)
end
Note: I commented out the File.write line that will create the HTML files.
Can you try this? It should be reading lines of file
f = File.new("name_of_file")
f.each_line { |line| puts line }
You can later save them as new file, more on that here:
How to create a file in Ruby
I am trying to pull out the meta description of a few webpages. Below is my code:
URL_List = ['https://digisapient.com', 'https://dataquest.io']
Meta_Description = []
for url in URL_List:
response = requests.get(url, headers=headers)
#lower_response_text = response.text.lower()
soup = BeautifulSoup(response.text, 'lxml')
metas = soup.find_all('meta')
for m in metas:
if m.get ('name') == 'description':
desc = m.get('content')
Meta_Description.append(desc)
else:
desc = "Not Found"
Meta_Description.append(desc)
Now this is returning me the below:
['Not Found',
'Not Found',
'Not Found',
'Not Found',
'Learn Python, R, and SQL skills. Follow career paths to become a job-qualified data scientist, analyst, or engineer with interactive data science courses!',
'Not Found',
'Not Found',
'Not Found',
'Not Found']
I want to pull the content where the meta name == 'description'. In case, the condition doesn't match, i.e., the page doesn't have meta property with name == 'description it should return Not Found.
Expected Output:
['Not Found',
'Learn Python, R, and SQL skills. Follow career paths to become a job-qualified data scientist, analyst, or engineer with interactive data science courses!']
Please suggest.
let me know if this works for you!
URL_List = ['https://digisapient.com', 'https://dataquest.io']
Meta_Description = []
meta_flag = False
for url in URL_List:
response = requests.get(url, headers=headers)
meta_flag = False
#lower_response_text = response.text.lower()
soup = BeautifulSoup(response.text, 'lxml')
metas = soup.find_all('meta')
for m in metas:
if m.get ('name') == 'description':
desc = m.get('content')
Meta_Description.append(desc)
meta_flag = True
continue
if not meta_flag:
desc = "Not Found"
Meta_Description.append(desc)
The idea behind the code is that it will iterate through all the item in metas, if a 'description' is found, it will set the flag to be True, thereby skipping the subsequent if-statement. If after iterating through metas and nothing is found, it will then append "Not Found" to Meta_Description.
Your result is actually what it's supposed to be.
Your current code
Let's have a look.
for url in URL_List:
For each page in your list,
metas = soup.find_all('meta')
you want all the <meta> tags (regardless of other attributes).
for m in metas:
For each <meta> tag, check if it's a <meta name="description">. If it is, save its content; otherwise, save "Not Found".
HTML <meta>
See more info on MDN's docs. In short, you can use <meta> for more meta needs, and as such, you can have multiple <meta> tags in your HTML document.
Your URLs
If you actually open up your URLs and have a look at the HTML, you'll see that you get
digisapient (1 <meta>)
<meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no" />
dataquest (a lot more <meta>s)
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Learn Python, R, and SQL skills. Follow career paths to become a job-qualified data scientist, analyst, or engineer with interactive data science courses!" />
<meta name="robots" content="index, follow" />
<meta name="googlebot" content="index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1" />
<meta name="bingbot" content="index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1" />
<meta property="og:locale" content="en_US" />
<meta property="og:type" content="website" />
<meta property="og:title" content="Learn Data Science Online and Build Data Skills with Dataquest" />
<meta property="og:description" content="Learn Python, R, and SQL skills. Follow career paths to become a job-qualified data scientist, analyst, or engineer with interactive data science courses!" />
<meta property="og:url" content="https://www.dataquest.io/" />
<meta property="og:site_name" content="Dataquest" />
<meta property="article:publisher" content="https://www.facebook.com/dataquestio" />
<meta property="article:modified_time" content="2020-05-14T22:43:29+00:00" />
<meta property="og:image" content="https://www.dataquest.io/wp-content/uploads/2015/02/dataquest-learn-data-science-logo.jpg" />
<meta property="og:image:width" content="1040" />
<meta property="og:image:height" content="520" />
<meta name="twitter:card" content="summary_large_image" />
<meta name="twitter:creator" content="#dataquestio" />
<meta name="twitter:site" content="#dataquestio" />
As you can see, the second website has many more such tags in its content, and for every one of those, you go through that last if/else statement; if it doesn't have a name="description", you save a "Not Found" in your results set.
Your actual results
To check what your program is doing at a time, and better understand the different values that the variables get over time, I think it's worthwhile to look up debugging and start doing it.
For a quick 'n dirty solution, try writing some messages to the screen as your program's execution progresses, e.g. Downloaded URL http://..., Found 4 <meta> tags, Found 0 <meta name="description".
I suggest you go over your program line by line, with the HTML in the other hand, and try to see what should happen.
Getting there
From your expected results, it seems that you don't actually care about the "Not Found"s, and you know up front that you always want the <meta name="description"> tags.
With CSS Selectors
You can try selecting DOM elements on your own in the browser, using document.querySelectorAll(). Open up your browser's developer tools / JavaScript console and type in, for instance
document.querySelectorAll("meta[name='description']")
to get all of the page's meta tags with a name attribute, which has the value of description. See more on CSS selectors.
Having this in mind is important, because
1. you get a better feel of what you're looking at / trying to do
1. you can actually use this type of selectors with BeautifulSoup, as well!
With BeautifulSoup
So you can move the check for the name attribute up in the query. Something like
metas = soup.find_all('meta', attrs={"name": "description"})
and that should give you only the tags which have name=description in them. This means that all the other <meta>s will be ignored.
Alternatively, you can keep the current query method (get all <meta>s) and just ignore them in the if/else statement, i.e. don't save them in your results list. If you want to know that it's actually doing something, but it just doesn't match up you required query, you could simply log a message, instead of saving the "Not Found".
how to replace text content of html tag in file and save them to another(some), file ?
Ex. there is a file index.html
<!DOCTYPE html>
<html>
<head>
</head>
<body>
<p itemprop="someprop">SOME BIG TEXT</p>
</body>
</html>
I need to replace the text "SOME BIG TEXT" in the "p" tag to "ANOTHER BIG TEXT"
from bs4 import BeautifulSoup
with open("index.html","r") as file:
fcontent=file.read()
sp=BeautifulSoup(fcontent,'lxml')
t='new_text_for_replacement'
print(sp.replace(sp.find(itemprop="someprop").text,t))
What am I doing wrong ?
Thank you
Use open() on the output file to write to it.
with open('index.html', 'r') as file:
fcontent = file.read()
sp = BeautifulSoup(fcontent, 'html.parser')
t = 'new_text_for_replacement'
# replace the paragraph using `replace_with` method
sp.find(itemprop='someprop').replace_with(t)
# open another file for writing
with open('output.html', 'w') as fp:
# write the current soup content
fp.write(sp.prettify())
If you want to replace just the inner content of the paragraph instead of the paragraph element itself, you can set the .string property.
sp.find(itemprop='someprop').string = t
The problem relies upon on the way you are searching for the criteria try changing the following code:
print(sp.replace(sp.find(itemprop="someprop").text,t))
to this:
print(sp.replace(sp.find({"itemprop":"someprop"}).text,t))
hopefully, this helps
(PS: based of your questionI'm assuming that you only have one thing to replace)
<div class="vcard" itemscope itemtype="http://schema.org/LocalBusiness">
<strong class="fn org" itemprop="name">Commercial Office Bangalore</strong><br/>
<span class="adr" itemprop="address" itemscope itemtype="httep:schema.org/PostalAddress">
<span class="street-address"itemprop="streetAddress">330, Raheja Arcade, 1/1 Koramangala Industrial Layout</span><br/>
<span class="locality"itemprop="addressLocality">Bangalore</span>
<span itemprop="postalCode"class="postal-code">560095</span><br/>
<span class="region"itemprop="addressRegion">Karnataka</span><br/>
<span class="country-name">India</span><br/>
<span class="tel id"="phone"itemprop="telephone">+91-80-41101360</span><br/>
<span class="geo"itemprop="geo"itemscope itemtype="http://schema.org/GeoCordinates">
<abbr class="latitude" property="latitude">12.936504</abbr>
<abbr class="longitude" property="longitude">77.6321344</abbr>
<meta itemprop="latitude" content="12.936504" />
<meta itemprop="longitude" content="77.6321344" />
</span>
</span>
I want to make lat and logitude invisible for user and visible for Google bot. What shall I do?
If you mean the Microdata (using Schema.org):
Just remove the abbr elements. You are already giving this data in meta elements (which can be used in the body), which is the correct way to provide data that should/can not be visible on the page:
<meta itemprop="latitude" content="12.936504" />
<meta itemprop="longitude" content="77.6321344" />
(Note that you were using property attributes, but they are not allowed in Microdata, only in RDFa.)
(Also note that you use http://schema.org/GeoCordinates but it should be http://schema.org/GeoCoordinates. And httep:schema.org/PostalAddress is also wrong.)
If you mean the Microformat (using hCard):
You could reuse the meta elements used with Microdata:
<meta class="latitude" itemprop="latitude" content="12.936504" />
<meta class="longitude" itemprop="longitude" content="77.6321344" />
But I’m not sure if all Microformat parsers support this.
The abbr elements with lat and lon can be set to display:none, which does exactly what you are asking for, hiding the content from humans, while still serving it up to bots:
abbr{display:none}
/** or **/
.latitude,
.longitude{display:none}
Although I can't honestly see what's wrapping them in your example. Ff the span with class .geo is the parent element, you could make that display:none instead.