Scraping Data Using Requests and Beautifulsoup - python-3.x

Question

I am trying to scrape data from this link. Where I want to first find all headings that are in bold.
I've achieved the above task using code below:
url = 'https://www.emirates.com/pk/english/help/covid-19/dubai-travel-requirements/tourists/'
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')
headers = []
for sib in soup.findAll('strong'):
headers.append([sib.text])
The problem is there is a bold text in li tag I don't want that as header. E.g. If you are flying from India, Pakistan, Nigeria or Bangladesh is considered as header I don't want that to be included in header as it is in li tag. How can I solve this?
Next part where I am stuck is that I want to scrape all text under these headers. To achieve that I've written the following code:
main_data = []
data_str = ''
for i in range(0, len(headers)):
target = soup.find(['h3', 'p'], text=headers[i])
for sib in target.find_next_siblings():
if sib.name == "strong":
break
else:
data_str = sib.text + "."
main_data.append([data_str])
Currently the output contains list of lists but each tag is made a list. Also the content and headers are repeating.
The expected output is a list of lists containing text scraped from under each header.
Example:
For header Passengers will need to do COVID‑19 PCR tests only if it is mandated by the country they are travelling to.
main_data[0] = Please check the requirements of the country you are travelling to. The travel regulations change frequently. You may need to take a COVID‑19 PCR test before you depart or another particular type of COVID‑19 test specified by your destination.
This is a list of authorised COVID‑19 test laboratories in Dubai where you can get tested before you travel to your destination.

Answer 1

Solution of the first part can be:
import requests
from bs4 import BeautifulSoup
url = 'https://www.emirates.com/pk/english/help/covid-19/dubai-travel-requirements/tourists/'
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')
headers = []
for sib in soup.select('p > strong'):
headers.append([sib.text])
for sib in soup.select('h3 > strong'):
headers.append([sib.text])
headers
Output:
[['Passengers will need to do COVID-19 PCR tests only if it is mandated by the country they are travelling to.'],
['Rapid COVID-19 testing at Dubai International airport for flights to China'],
['Special COVID-19 PCR test rates for Emirates passengers'],
['Requirements for all passengers arriving in Dubai'],
['Indian Nationals with a normal passport who are travelling to or from India via Dubai can obtain a visa on arrival in Dubai for a maximum stay of 14 days provided they:'],
['Test on arrival'],
['Transiting in Dubai'],
['Test exemptions:'],
['COVID-19 testing laboratories:'],
['Arriving passengers'],
['Vaccination certificate verification'],
['Before you book'],
['Before you travel'],
['When you arrive']]
Solution of the second part:
main_data = []
for i in range(0, len(headers)):
target = soup.find(['h3', 'p'], text=headers[i])
text = ''
for sib in target.find_next_siblings():
if sib.select_one('p > strong') is not None or sib.select_one('h3 > strong') is not None:
break
else:
text += sib.text
main_data.append([text])
main_data
Output:
[['Please check the requirements of the country you are travelling to. The travel regulations change frequently. You may need to take a COVID-19 PCR test before you depart or another particular type of COVID-19 test specified by your destination.This is a list of authorised COVID-19 test laboratories in Dubai\ufeff where you can get tested before you travel to your destination.'],
['All passengers, except children under the age of 1, who are travelling to China must have a negative rapid COVID-19 test certificate before travel. You must report to the check-in counter 5 hours before your flight and take this test at Dubai International airport at Emirates Terminal 3 departure area, next to Costa. For further information please refer to the travel requirements for China.'],
['Emirates has expanded its medical partnerships to offer all passengers exclusive home or office COVID-19 PCR testing rates at the following centres:Al Tadawi Medical CentreLocated at Al Masood building, Airport Road, Port Saeed area, Deira.The test costs AED 130 per person. Home or office testing within Dubai costs AED 240 per person. Test results will be available within 24 hours.Prime Medical CentresLocations in Dubai:\nAl Qusais Branch, Damascus StreetPremier Diagnostic and Medical Center, Salah Al Din Street\xa0Prime Corp Medical Center, Salah Al Din Street, DeiraSheikh Zayed Branch, Sheikh Zayed Street, near Noor Islamic BankPrime Specialist Medical Center Sharjah Branch, King Faisal St, Al MajazAjman Branch, Grand Mall, Sheikh Khalifa StThe test costs AED 150 per person. Home or office testing within Dubai for a minimum of two passengers is also available at AED 240 per person. Test results will be available within 24 hours.'],
["All passengers travelling to Dubai from any point of origin (GCC countries included) must hold a negative COVID-19 RT-PCR test certificate for a test taken no more than 72 hours before departure, except for travel from Bangladesh, Ethiopia, India, Nigeria, Pakistan, Sri Lanka, South Africa, Uganda, Vietnam, Zambia (for which specific requirements are stated above). Please see the requirements for travel from India below.The certificate must be a Reverse Transcription-Polymerase Chain Reaction (RT-PCR) test. Other test certificates including antibody tests, NHS COVID Test certificates, Rapid PCR tests and home testing kits are not accepted in Dubai. Travellers must bring an official printed or digital certificate in English or Arabic to check in – SMS certificates are not accepted. PCR certificates in other languages are acceptable if they can be validated at the originating station.COVID-19 RT-PCR test certificates must be issued by an authorised facility in the passenger's departure country. Certificates that have already been presented for travel to another destination can't be used for re-entry even if they are still within the validity period.For passengers arriving from the following countries, it is mandatory that the COVID-19 PCR report includes a QR code linked to the original report for verification purposes. The QR code must be presented at check-in and to representatives of the Dubai Health Authority (DHA) upon arrival in Dubai airports: Indonesia, Sudan, Lebanon, Egypt and Ethiopia."],
['have a visitor visa or a green card issued by the Unites States, ora residence visa issued by the United Kingdom or Europe unionThe visa issued by United States, United Kingdom or Europe union has to be valid for a minimum of 6 months'],
['Passengers arriving in Dubai from the following countries will be required to take another COVID-19 PCR test on arrival at Dubai International airport:Afghanistan, Angola, Argentina, Azerbaijan, Bahrain, Bangladesh, Bosnia & Herzegovina, Brazil, Cambodia, Chile, Croatia, Cyprus, Djibouti, Egypt, Eritrea, Ethiopia, Georgia, Ghana, Greece, Guinea, Hungary, India, Indonesia, Iran, Iraq, Israel, Ivory Coast, Jordan, Kenya, Kuwait, Kyrgyzstan, Lebanon, Malta, Moldova, Montenegro, Morocco, Myanmar, Nepal, Pakistan, Poland, Philippines, Qatar, Rwanda, Russia, Senegal, Slovakia, Somaliland, Somalia, South Africa, South Sudan, Sudan, Syria, Tajikistan, Tanzania, Thailand, Tunisia, Turkey, Turkmenistan, Uganda, Ukraine, Uzbekistan, Zimbabwe.'],
['All transit passengers must complete all the requirements of their final destination.Transit passengers from the following countries must present a negative COVID-19 PCR test certificate for a test taken no more than 72 hours before departure:\xa0Bangladesh, Ethiopia, India, Nigeria, Pakistan, Sri Lanka, South Africa, Uganda, Vietnam, Zambia, IndonesiaAll other transit passengers are not required to present this certificate unless it is mandated by their final destination.'],
["UAE nationals are exempt from taking a COVID-19 PCR test before departing for Dubai. They must be tested on arrival in Dubai, irrespective if they are holding a valid negative COVID-19 RT-PCR certificate from the point of origin.\n\nThis is also applicable for:\nPassengers accompanying a 1st degree UAE nationals' relative or domestic workersDomestic workers escorting a UAE national sponsor during travel.Children under the age of 12 and passengers who have a moderate or severe disability are exempt from taking a COVID-19 RT-PCR test.\nModerate or severe disability includes neurological disorders and intellectual or developmental disabilities. For example: Acute spinal cord injury, Alzheimer's disease, Amyotrophic lateral sclerosis (ALS), Ataxia, Autism spectrum, Bell's palsy, Brain tumours, Cerebral aneurysm, Cerebral palsy, Down Syndrome, Epilepsy and seizuresAll other passengers, including those who are visually impaired, hearing impaired or physically challenged must hold a negative COVID-19 RT-PCR test certificate as per the requirements.There may be specific test exemptions in your country of origin and final destination. Please check the requirements before you travel."],
['The UAE government has specified designated laboratories.\ufeff You can either use the recommended laboratories in the list or any trusted and certified laboratories in your country of origin to get your COVID-19 RT-PCR test.If you are flying from India, Pakistan, Nigeria or Bangladesh , you must get your certificate from one of the labs listed in the designated laboratories document to be accepted on the flight.'],
['Passengers who are planning to travel to Abu Dhabi must comply with the following protocols in place at all Abu Dhabi borders. These procedures may affect travel time.Effective 5 September 2021, Abu Dhabi authorities have revised the rules and updated travel procedures for UAE citizens and residents as well as visitors entering Abu Dhabi.Vaccinated travellersVaccinated travellers from green list destinations must take a COVID-19 PCR test on arrival and on day 6 after arrival but do not have to undergo quarantine.When arriving from other destinations (non-green), they must take a COVID-19 PCR test on arrival and on days 4 and 8 after arrival but do not have to undergo quarantine.The protocol applies to fully vaccinated UAE citizens and residents as well as visitors, which is also documented in the Alhosn App.Unvaccinated travellersUnvaccinated citizens, residents and visitors arriving into Abu Dhabi from green list destinations must take a COVID-19 PCR test on arrival and on days 6 and 9 after arrival but do not have to undergo quarantine.When arriving from other destinations (non-green), they must take a COVID-19 PCR test on arrival, quarantine for 10 days, wear a medically approved wristband and take another COVID-19 PCR test on day 9 of quarantine.To be considered fully vaccinated, individuals must have received two doses of the same vaccine at least 14 days before departure.'],
['Before departure, visitors need to register in the Register Arrivals section of the Federal Authority for Identity and Citizenship (ICA) app, complete the register arrivals form and upload an international vaccination certificate. Visitors will then receive an SMS including a link to download the Alhosn app.Upon arrival in Abu Dhabi, visitors will receive a Unified Identification Number (UID) either at the airport or via ICA app or website. Visitors will then need to download and register on the Alhosn app using the UID and phone number used for ICA registration or when taking a COVID-19 PCR test in the UAE.Visitors will receive a one-time password (OTP) to complete the Alhosn app registration process. Alhosn app allows users to check status, vaccination information, test results and travel test requirements and use a live QR code.These tests and processes are a legal requirement and those failing to follow this process are liable for fines.Find out where to get tested in Dubai before you enter Abu Dhabi\ufeff.'],
['Check if you need a visa.\ufeff Depending on your nationality you can get a visa on arrival, or you can apply for your prearranged visit visa from Dubai Immigration before you travel.'],
['GDRFA\ufeff or ICA\ufeff approval is not required for tourists travelling to the UAE.Passengers arriving from the following countries must follow specific protocols:Bangladesh, Ethiopia, India, Nigeria, Pakistan, Sri Lanka, South Africa, Uganda, Vietnam, ZambiaRequirements for passengers from these countries:A valid negative COVID-19 PCR test certificate with a QR code issued within 48 hours prior to departure from an approved health facilityA rapid PCR test report with a QR code for a test conducted at the departure airport within six hours of departureFor passengers travelling to Dubai as their final destination from Bangladesh, Nigeria, Vietnam and Zambia, travel is currently not possible as there are no rapid PCR testing facilities at the airport.'],
['You may need to take another COVID-19 PCR test on arrival. If you take a test at the airport, you must remain in your hotel or residence until you receive the test result.If the test result is positive, you will be required to undergo isolation and follow the Dubai Health Authority guidelines.You must also download the COVID19 – DXB Smart App iOS\ufeff-Android\ufeff']]