Get a list of every Layer, in every Service, in every Folder in an ArcGIS REST endpoint - python-3.x

Question

I have two ArcGIS REST endpoints for which I am trying to get a list of every layer:
https://rdgdwe.sc.egov.usda.gov/arcgis/rest/services
https://services1.arcgis.com/RLQu0rK7h4kbsBq5/ArcGIS/rest/services
These are not my organization's endpoints so I don't have access to them internally. At each of these endpoints there can be folders, services, and layers, or just services and layers.
My goal is to get a list of all layers. So far I have tried:
endpoints=(["https://rdgdwe.sc.egov.usda.gov/arcgis/rest/services",
"https://services1.arcgis.com/RLQu0rK7h4kbsBq5/ArcGIS/rest/services"])
for item in endpoints:
reqs = requests.get(item, verify=False)
# used this verify because otherwise I get an SSL error for endpoints[0]
soup =BeautifulSoup(reqs.text, 'html.parser')
layers = []
for link in soup.find_all('a'):
print(link.get('href'))
layers.append(link)
However this doesn't account for the variable nested folders/services/layers or services/layer schemas, and it doesn't seem to be fully appending to my layers list.
I'm thinking I could also go the JSON route and append ?f=psjon . So for example:
https://rdgdwe.sc.egov.usda.gov/arcgis/rest/services/?f=pjson would get me the folders
https://rdgdwe.sc.egov.usda.gov/arcgis/rest/services/broadband/?f=pjson would get me all the services in the broadband folder
and
https://rdgdwe.sc.egov.usda.gov/arcgis/rest/services/broadband/CDC_5yr_OpioidOverDoseDeaths_2016/MapServer?f=pjson would get me the CDC_OverDoseDeathsbyCounty2016_5yr layer in the first service (CDC_5yr_OpioidOverDoseDeaths_2016) in the broadband folder.
Any help is appreciated. I put this here vs in the GIS stack exchange as it seems a more python question than geospatial.

Answer 1

I don't really agree this is a Python question because there doesn't seem to be any issue with how to use the various Python libraries. The main issue appears to be how do you work with Esri's REST API. Seeing that Esri is very much a GIS company and their REST API is very much a GIS API, I think GIS StackExchange would have been a better forum for the question.
But, since we are here now....
If you are going to continue working with Esri's REST API with Python, I strongly encourage you to read up on Esri's ArcGIS API for Python. At its core, the ArcGIS API for Python is a Python wrapper for working with Esri's REST API. Unless someone has very basic needs, rolling one's own Python code for Esri's REST API isn't time well spent.
If you are set on rolling your own, I strongly encourage you to read Get started -- ArcGIS REST APIs | ArcGIS Developers. The documentation describes the structure of the REST API, syntax, and it includes some examples.
The following isn't pretty, it is meant more to help you connect the dots when reading Esri's documentation. That said, it will give you a list of Map Services on an ArcGIS Server site and the layers for those services.
import json
import requests
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
services = {}
services_endpoint = "https://fqdn/arcgis/rest/services"
req = requests.get(f"{services_endpoint}?f=json", verify=False)
svcs_root = json.loads(req.text)
for fld in svcs_root['folders']:
req = requests.get(f"{services_endpoint}/{fld}?f=json", verify=False)
svcs_fld = json.loads(req.text)
for svc in svcs_fld['services']:
if svc['type'] not in ('MapServer'): continue
req = requests.get(f"{services_endpoint}/{svc['name']}/{svc['type']}?f=json", verify=False)
svc_def = json.loads(req.text)
services.update({svc['name']:{'type':svc['type'], 'layers':svc_def['layers']}})
for svc in svcs_root['services']:
if svc['type'] not in ('MapServer'): continue
req = requests.get(f"{services_endpoint}/{svc['name']}/{svc['type']}?f=json", verify=False)
svc_def = svc = json.loads(req.text)
services.update({svc['name']:{'type':svc['type'], 'layers':svc_def['layers']}})

Answer 2

As part of developing GISsurfer (https://gissurfer.com) I was faced with that exact problem but for any ArcGIS server that did not require login creds. My solution was to write PHP code to 'walk the tree' to find all services.