How can I change remote controller of FireFox in Selenium Python? - python-3.x

Question

When I FireFox is loaded through Selenium, my browser is under remote controller and a bot image show in URL section in my browser. For deal with this problem I changed User-Agent by this code:
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override", "whatever you want")
driver = webdriver.Firefox(profile)
User-Agent was changed successfully but, bot image in URL section of my browser was remained.
Would you help me, please?
I used this URL for changing User-Agent:
Change user agent for selenium driver
My whole code is:
MainLink="https://blog.feedspot.com/iot_blogs/"
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
caps = DesiredCapabilities.PHANTOMJS
caps["phantomjs.page.settings.userAgent"] = "whatever you want"
driver = webdriver.Firefox()
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override", "whatever you want")
driver = webdriver.Firefox(profile)
agent = driver.execute_script("return navigator.userAgent")
print(agent)
driver.get(MainLink)

Answer 1

Your code is a little confusing. You don't need to use both phantomjs and firefox as the driver for selenium. Which one are you going to use?
As I understood, you would like to avoid being detected from the page you're interacting with. This is usually a greater concern if you're operating with a headless browser, which is the case when using phantomjs but not when using firefox without explicitly telling it to run in this mode, which apparently is your case.
If you are in fact having issues of this nature, there are many ways to try to mitigate this, starting from changing the user agent, as mentioned by you. Assuming you want to go with firefox, A code example would be:
from selenium import webdriver
profile = webdriver.FirefoxProfile()
user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'
profile.set_preference("general.useragent.override", user_agent)
driver = webdriver.Firefox(profile)
MainLink="https://blog.feedspot.com/iot_blogs/"
driver.get(MainLink)
In addition, you could set a different user agent each time you make a request, combining with changing the ip address from where the requests are originated, of course, but this is besides the point...
Hope this helps...

Answer 2

Depending on how "deep" you want to go, you could also add cookies copied from a trusted client:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("http://www.example.com")
# Adds the cookie into current browser context
driver.add_cookie({"name": "key", "value": "value"})
See also: https://www.selenium.dev/documentation/en/support_packages/working_with_cookies/

Answer 3

I could run my code without loading page or in headless way (headless browser).
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.add_argument("--headless")
driver = webdriver.Firefox(firefox_options=options)
print("Firefox Headless Browser Invoked")
driver.get('https://blog.feedspot.com/iot_blogs/')
s=driver.find_element_by_xpath("""/html/body/div[1]/div[2]/div/div/div[1]/article/div[1]/h1""")
print(s.text)