I have been exploring the YouTube Data API. The premise of my project is simple: using the API, authenticate (yes, I have the credentials for the account) and then simply retrieve the list of all my videos, public and private.
I have been able to accomplish this successfully, except for the fully automated part. I have used code from various sources and when I run it on the command line, it provides me a link to be used in a browser so that the authorization takes place.
It looks something like this:
Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=7932902759886-cb8ai84grcqshe24nn459ka46uh45ssj.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fyoutube.readonly&state=zNVvgEyO47nmacvdEEAhDsQipY194k&prompt=consent&access_type=offline&code_challenge=aF7uTCghjwgwjg49o3fgiIU-_ryK19rDeX4l1uzr37w&code_challenge_method=S256
Enter the authorization code:
....
Here's a snippet of my python code:
import google_auth_oauthlib.flow
import googleapiclient.discovery
import googleapiclient.errors
...
...
# Get credentials and create an API client
flow = google_auth_oauthlib.flow.InstalledAppFlow.from_client_secrets_file(
client_secrets_file, scopes)
credentials = flow.run_console()
youtube = googleapiclient.discovery.build(
api_service_name, api_version, credentials=credentials)
## MAKE youtube SEARCH REQUEST
last_date = '2018-10-01T00:00:00Z'
request = youtube.search().list(
part="snippet",
forMine=True,
maxResults=50,
order="date",
type="video"
)
all_items = []
response = request.execute()
My question here is the following: Is it possible to programatically perform the authorization so that the app can run standalone and not have to wait for this user action (to literally copy the URL from CMD, visit to get the token, and the copy and paste the token again)? I'd like to schedule this and therefore would like it to run and authenticate without human intervention. Is this possible at all? If so, can someone please point me to some working examples and/or other resources to help me get there? Thanks a million.
# -*- coding: utf-8 -*-
# Sample Python code for youtube.channels.list
# See instructions for running these code samples locally:
# https://developers.google.com/explorer-help/guides/code_samples#python
#!/usr/bin/python3.7
import os
import pickle
import google_auth_oauthlib.flow
import googleapiclient.discovery
import googleapiclient.errors
scopes = ["https://www.googleapis.com/auth/youtube.readonly"]
client_secrets_file = "client_secret.json"
api_service_name = "youtube"
api_version = "v3"
def main():
# Disable OAuthlib's HTTPS verification when running locally.
# *DO NOT* leave this option enabled in production.
os.environ["OAUTHLIB_INSECURE_TRANSPORT"] = "1"
# Get credentials and create an API client
flow = google_auth_oauthlib.flow.InstalledAppFlow.from_client_secrets_file(
client_secrets_file, scopes)
youtube = get_authenticated_service()
request = youtube.channels().list(
part="contentDetails",
mine=True
)
response = request.execute()
print(response)
def get_authenticated_service():
if os.path.exists("CREDENTIALS_PICKLE_FILE"):
with open("CREDENTIALS_PICKLE_FILE", 'rb') as f:
credentials = pickle.load(f)
else:
flow = google_auth_oauthlib.flow.InstalledAppFlow.from_client_secrets_file(client_secrets_file, scopes)
credentials = flow.run_console()
with open("CREDENTIALS_PICKLE_FILE", 'wb') as f:
pickle.dump(credentials, f)
return googleapiclient.discovery.build(
api_service_name, api_version, credentials=credentials)
if __name__ == "__main__":
main()
The Credentials instance from credentials = flow.run_console() has a built-in functionality to refresh token.
It'll will refresh the token when a request being execute if needed.
Therefore you can save the credentials object into pickle, and read it back when need it
A few alteration on Google python sample code:
def get_authenticated_service():
if os.path.exists(CREDENTIALS_PICKLE_FILE):
with open(CREDENTIALS_PICKLE_FILE, 'rb') as f:
credentials = pickle.load(f)
else:
flow = InstalledAppFlow.from_client_secrets_file(CLIENT_SECRETS_FILE, SCOPES)
credentials = flow.run_console()
with open(CREDENTIALS_PICKLE_FILE, 'wb') as f:
pickle.dump(credentials, f)
return build(API_SERVICE_NAME, API_VERSION, credentials = credentials)
copied from https://developers.google.com/identity/protocols/OAuth2InstalledApp
Step 3: Google prompts user for consent
In this step, the user decides whether to grant your application the requested access. At this stage, Google displays a consent window that shows the name of your application and the Google API services that it is requesting permission to access with the user's authorization credentials. The user can then consent or refuse to grant access to your application.
Your application doesn't need to do anything at this stage as it waits for the response from Google's OAuth 2.0 server indicating whether the access was granted. That response is explained in the following step.
Where this is important:
At this stage, Google displays a consent window that shows the name of your application and the Google API services that it is requesting permission to access with the user's authorization credentials.
So, at least as I interpret it, what you want to do should not be done for security reasons.
However: you can "simulate" a browser by how ever python have libs for do such. On the other hand: Once you got the auth-token you can re-use it instead of request a new token each time. I couldn't find it in provided doc on GitHub, but Java as example supports to store an obtained token along with its refresh token so it can be reused once obtained and auto-refreshed. Maybe python provides some way to store the obtained token (check if it contains a refresh token) and re-load it. Also: if you load such token, first you have to do is to refresh it before using it. Java provieds a way to just save a refresh token instead of the whole auth-token wich can be used in a later run to automatic obtain a new auth-token.
As response is a JSON maybe you could build some yourself if the lib doesn't already offer this.
// edit
In addition from https://github.com/googleapis/google-auth-library-python/blob/master/google/oauth2/credentials.py
There are methods to load a credential object either from an "authorized user info" (wich I also somewhere found can be loaded from file) or to load it directly from file. So, I guess you just have to figure out how to store the token. As doc says for from_authorized_user_file:
Creates a Credentials instance from an authorized user json file.
I guess that means you just have to save the token response you get after the initial authorization was done.
Related
I'm using google api to integrate some google sheet data into my app. I have a server where I can only connect via SSH, as it has no user interface. When I start my app, it gives me this message:
Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=XXX-XXX.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A57731%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fspreadsheets.readonly&state=XXX&access_type=offline
Since I don't have a UI on my server, how can I authorize the application? I've just started using the google api, I still can't figure out how to go about it.
The only thing I've tried and that works is to run the code from a windows pc for example and then transfer the token.json file to my server. The problem is that after 30 days it expires and I have to redo the whole procedure again.
Is there something I'm doing wrong? Here is my code:
from __future__ import print_function
from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError
import os.path
# If modifying these scopes, delete the file token.json.
SCOPES = ['https://www.googleapis.com/auth/spreadsheets.readonly']
# The ID of a sample spreadsheet.
SPREADSHEET_ID = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
# Google API credentials
googleCredentials = None
def main():
print("Loadin google API...")
global googleCredentials
if os.path.exists("./json/token.json"):
googleCredentials = Credentials.from_authorized_user_file("./json/token.json", SCOPES)
# If there are no (valid) credentials available, let the user log in.
if not googleCredentials or not googleCredentials.valid:
if googleCredentials and googleCredentials.expired and googleCredentials.refresh_token:
googleCredentials.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file("./json/credentials.json", SCOPES)
googleCredentials = flow.run_local_server(port=0)
# Save the credentials for the next run
with open("./json/token.json", 'w') as token:
token.write(googleCredentials.to_json())
if __name__ == "__main__":
main()
The only thing I've tried and that works is to run the code from a windows pc for example and then transfer the token.json file to my server. The problem is that after 30 days it expires and I have to redo the whole procedure again.
It sounds like you have done exactly what you should do. The token.json file should be just fine stored on your machine. The question is why is your refresh token expireing.
If your app is still in the testing phase the refresh token will expire after seven days. The solution for that is to go to google cloud console for your app then to the Oauth2 consent screen and set it to production the next time you create a refresh token it will not expire.
I'm a bit curious as to why you say this lasts 30 days which me think you might be getting a new refresh token back every time it refreshes your access token and its not being saved but that's not supposed to happen with the python library its supposed to handle that I will have to test it.
If you are only connecting to this single sheet, Have you considered using a service account instead?
service account
Just make sure to share the sheet with the service accounts email address, just like you would any other user via the google drive web app. It will have access to your sheet without any user intervension.
credentials = ServiceAccountCredentials.from_json_keyfile_name(
KEYFILE,
scopes=SCOPES,
)
How to create service account credentials
Should you be using a Google service account
Token.json explained
Below is the contense of a token.json file, this is exactly what an authorization server returns to you as the final step in the oauth flow.
The first on Token is actually your access token, the expire says when that access token will expire. When your code loads it will check if the access token in the file has expired if it has it will request a new one using the refresh token. look for: googleCredentials.expired
{
"token": "[Redacted Access Token]",
"refresh_token": "[Redacted refresh Token]",
"token_uri": "https://oauth2.googleapis.com/token",
"client_id": "[REDACTED]",
"client_secret": "[REDACTED]",
"scopes": [
"https://www.googleapis.com/auth/calendar.readonly"
],
"expiry": "2022-10-25T12:06:40.318779Z"
}
I followed the quickstart example to integrate my django app with google calendar. The difference from quickstart to my situation is that i just want to generate a URL and send it back to my user, through
from google_auth_oauthlib.flow import InstalledAppFlow
SCOPES = ['https://www.googleapis.com/auth/calendar']
flow = InstalledAppFlow.from_client_secrets_file(f"{PATH_TO_FILE}/{CLIENT_SECRET_FILE}", SCOPES)
(auth_url, state) = flow.authorization_url()
if is_dev():
auth_url += '&redirect_uri=http%3A%2F%2Flocalhost%3A43759%2F'
print(auth_url)
(OBS: I added this is_dev option, because no redirect_uri was not considered)
I get this printed URL and get this steps:
1- The URL from auth_url printed when i ran the program
2- After choosing my user
3- and BAM, I cant proceed (i am redirected to localhost:47759 and cant access)
What should I do?
we wen't through one solution, that 3 steps are important to talk about.
1- Create a new credential on Google Cloud, OAuth 2.0 Client ID for Web Application, as js origins with my local url, and another redirect URL authorized (this redirect solved the number 3 error for the question)
2- Also I read some examples about, and to get the user authorization, we send him an URL, if everything goes ok, he is redirected to our endpoint described above
from google_auth_oauthlib.flow import InstalledAppFlow
SCOPES = ['https://www.googleapis.com/auth/calendar']
flow = InstalledAppFlow.from_client_secrets_file(f"{PATH_TO_FILE}/{CLIENT_SECRET_FILE}", SCOPES)
flow.redirect_uri = URL_SAVED_ON_STEP_1
(auth_url, state) = flow.authorization_url()
print(auth_url)
3- And to the URL receiving my code, was necessary an endpoint where we could save the user credential and use it if we wanted to add an event on the user calendar
flow = InstalledAppFlow.from_client_secrets_file(f"{CONFIG_FILES_PATH}/{CLIENT_SECRET_FILE}", SCOPES)
flow.redirect_uri = URL_SAVED_ON_STEP_1
flow.fetch_token(authorization_response=request.url)
creds = flow.credentials
with open(f"{CONFIG_FILES_PATH}/token.json", 'w') as token:
token.write(creds.to_json())
So we can let any user share their calendar and we can manage as they allow
Working with google-api-python-client for the first time, I'm trying to generate a "link" in my authorization flow that I can pass to a user for them to allow the app to access their calendar, then I need google to pass-back the token to my app.
Currently I have something like this (basically the getcredentials() function from the quickstart demo, with user-specific tokens and WebApplication credentials.json):
def find_creds(user_id):
creds = None
token_pickle = f'./credentials/{user_id}.token.pickle'
if os.path.exists(token_pickle):
with open(token_pickle, 'rb') as token:
creds = pickle.load(token)
if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
'OAUTH.json', SCOPES) # Google WebApplication.json for OAUTH
creds = flow.run_local_server(port=0)
with open(token_pickle, 'wb') as token:
pickle.dump(creds, token)
return creds
This would work great if the user was running my application locally, however the issue is it's prompting on the server (vm) for a login, rather than passing the request to my users. No bueno.
The users are accessing my application through another application (which I don't control), so I can't really serve them a page to authorize the app - though I could pass them a URL/link to click.
This introduces a few new hurdles since the user isn't logging in locally, so I can't just "save" their authorization token.
The Authorization "flow" I'm trying to achieve should be (I think) something like this:
Pass the user a google authorization URL (I'm not sure how to generate this url/link, though I think it can be done with the google_auth_oauthlib.flow.InstalledAppFlow class. Maybe using authorization.url?)
User Authorizes app to access limited scope (calendar)
Google returns the users token back to my app (I guess this will need to be done via a return URi? So I think my server will need to run apache and have a listener running to collect/store credentials accordingly)
In tackling that first step, I'm already getting stuck though. I suspect that my flow object needs to change but I'm having a difficult time finding documentation on InstalledAppFlow:
Does it sound like I'm on the right track here? Any help/tips (or documentation) on InstalledAppFlow or google.oauth2.credentials class would be helpful too.
I've read through google-auth-library-python so far without figuring it out.
[UPDATE]
I have implemented the below solution but am running into problems when running as a service due to the manual intervention required from the print output.
Any other suggestions for a solution would be most welcome :(
[ORIGINAL]
I have been looking for a mail daemon type solution for a flask application. I'd like to integrate with Auzure rather than run a local mail server, so the MS Graph API looked to be a great choice. I then came across O365 which looked to be a perfect solution.
So I setup some testing and it works great, I even tried setting up the 2 step authentication as an option for login identity (Web app based authentication interface). This works but is not something I want for this project. I would just like the flask app to be able to send emails using Azure (app invites, password resets etc, various mail automations). Moving on...
I found the following authentication method to work perectly well and I dont need to handel any tokens and refreshing. However I cant find anything in the docs to suggest I'd be able to authentication this way but not print() the URL but redirect the user. Id like to be able to then capture the redirection in a route flow.
Have I missed something here?
from O365 import Account
credentials = ('CLIENT-ID', 'CLIENT-SECRET')
account = Account(credentials)
if not account.is_authenticated:
account.authenticate(scopes=['basic', 'message_all'])
print('Authenticated!')
# A URL is printed if not authed, you would then be required to navigate to this URL,
# give consent to the app then paste the returned URL back into the console.
# I want to try avoid this direct console approach.
Update:
As no one responded to my calls for help, I thought this might be useful to some.
I resolved this by using the two-stage authentication flow and TokenBackendStorage; details are here.
I collected Azure App details via a web front-end. It supplies instructions for configuring the application in Azure and captures relevant details to setup token-based access.
My issues were threefold. I wasn't setting permissions correctly (delegate vs application), I wasn't reading the documentation thoroughly enough, so was implementing a mishmash of auth methods.
This will need refining as its currently POC, storage of the token, for instance, should be encrypted either to disk or within a table on the DB and permissions should be set appropriately. Following the principle of least privilege.
Example authentication two-stage flow:
#app.route('/mail-config-step-one', methods=['GET'])
def mail_config_step_one():
if request.method == 'GET':
scopes = ['message_all']
try:
mail_data = db.session.query(mail_conf).one()
except:
mail_data = None
if mail_data :
application_id = mail_data.client_id
application_secret = mail_data.client_s
web_redirect = mail_data.web_redir
callback = web_redirect
credentials = (application_id, application_secret)
token_backend = FileSystemTokenBackend(token_path='./', token_filename='o365_token.txt')
account = Account(credentials, scopes=scopes, token_backend=token_backend, tenant_id='TENANT ID')
url, state = account.con.get_authorization_url(redirect_uri=callback)
mail_conf.query.filter_by(id=1).update(dict(auth_s=state))
print(url)
db.session.commit()
return redirect(url)
else:
return redirect(url_for('login'))
else:
return redirect(url_for('login'))
#app.route('/mail-config-step-two/', methods=['GET', 'POST'])
def mail_config_step_two():
scopes = ['message_all']
mail_data = db.session.query(mail_conf).one()
application_id = mail_data.client_id
application_secret = mail_data.client_s
web_redirect = mail_data.web_redir
credentials = (application_id, application_secret)
token_backend = FileSystemTokenBackend(token_path='./', token_filename='o365_token.txt')
account = Account(credentials, scopes=scopes, token_backend=token_backend, tenant_id='TENANT ID')
# retreive the state saved in mail-config-step-one
saved_state = mail_data.auth_s
# rebuild the redirect_uri used in auth_step_one
callback = web_redirect
result = account.con.request_token(request.url, state=saved_state, redirect_uri=callback)
# if the result is True, then authentication was successful
# and the auth token is stored in the token backend
if result:
flash('MailConfigurationForm: Authentication Was Successful!', 'success')
return redirect(url_for('configuration'))
else:
logger.error('mail_config_step_two failed: not post request')
return redirect(url_for('configuration'))
I am working on an application to pull the YouTube metrics using the YouTube analytics API. This is the first time I am working with google auth flow to authenticate my app so that it can silently pull the report.
I am following the below Google published article to work out the same:
https://developers.google.com/youtube/reporting/guides/authorization/server-side-web-apps
While following this article I am unable to figure out how to redirect the user to the auth_uri and obtain the auth_code.
Below is the code I have written for the auth flow so far:
API_SERVICE_NAME = 'youtubeAnalytics'
API_VERSION = 'v2'
CLIENT_SECRETS_FILE = 'C:/Users/Tushar/Documents/Serato_Video_Intelligence/client_secret_youtube.json'
def get_service():
global auth_code
global auth_uri
flow = client.flow_from_clientsecrets(
CLIENT_SECRETS_FILE,
scope='https://www.googleapis.com/auth/yt-analytics.readonly',
redirect_uri = "http://localhost:8080")
flow.params['access_type'] = 'offline'
flow.params['include_granted_scopes'] = True
auth_uri = flow.step1_get_authorize_url()
credentials = flow.step2_exchange(auth_code)
http_auth = credentials.authorize(httplib2.Http())
return build(API_SERVICE_NAME, API_VERSION, http=http_auth)
def execute_api_request(client_library_function, **kwargs):
response = client_library_function(
**kwargs
).execute()
if __name__ == '__main__':
# Disable OAuthlib's HTTPs verification when running locally.
# *DO NOT* leave this option enabled when running in production.
#os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = '1'
youtubeAnalytics = get_service()
execute_api_request(
youtubeAnalytics.reports().query,
ids='channel==XXXXXXXXXXXXXX',
startDate='2019-04-12',
endDate='2019-08-13',
filters= 'video==XXXXXXXXXXXXXX',
metrics='audienceWatchRatio,relativeRetentionPerformance',
dimensions='elapsedVideoTimeRatio'
)
When I run this code I get an error
File "C:\Users\Tushar\Documents\Serato_Video_Intelligence\youtube_auth_testing.py", line 65, in <module>
youtubeAnalytics = get_service()
File "C:\Users\Tushar\Documents\Serato_Video_Intelligence\youtube_auth_testing.py", line 40, in get_service
credentials = flow.step2_exchange(auth_code)
NameError: name 'auth_code' is not defined
I have gone through the articles on Stack Overflow and Google but have been unable to figure out what to do. I certainly know there are couple of steps that I am missing but I am unable to resolve it.
You want to retrieve the access token with the authorization process of OAuth2.
You want to achieve this using oauth2client with Python.
If my understanding is correct, how about this modification?
Authorization flow:
When the access token is retrieved by the authorization process of OAuth2, at first, it is required to authorize the scopes by own browser. When the scopes are authorize, the authorization code can be retrieved. Using this code, the refresh token and access token can be retrieved.
Patten 1:
When your current script is modified, it becomes as follows. In this modification, please modify get_service().
Modified script:
import httplib2
from apiclient.discovery import build
from oauth2client import client
def get_service():
flow = client.flow_from_clientsecrets(
CLIENT_SECRETS_FILE,
scope='https://www.googleapis.com/auth/yt-analytics.readonly',
redirect_uri='urn:ietf:wg:oauth:2.0:oob')
flow.params['access_type'] = 'offline'
auth_uri = flow.step1_get_authorize_url()
print('Please go to this URL: {}'.format(auth_uri))
auth_code = input('Enter the authorization code: ')
credentials = flow.step2_exchange(auth_code)
http_auth = credentials.authorize(httplib2.Http())
return build(API_SERVICE_NAME, API_VERSION, http=http_auth)
Or (This is from the sample script of Reports: Query.)
def get_service():
flow = InstalledAppFlow.from_client_secrets_file(CLIENT_SECRETS_FILE, SCOPES)
credentials = flow.run_console()
return build(API_SERVICE_NAME, API_VERSION, credentials = credentials)
When you run the script, Please go to this URL: ### is shown in the terminal. Please copy and paste it to your browser, and please authorize the scopes. Then, please retrieve the authorization code, and please paste it to the terminal. By this flow, the access token can be retrieved.
In above script, when the script is run, the authorization process is required to be done every time. So if you don't want to do this, how about the following pattern 2?
Patten 2:
In this pattern, when the script is run, the browser is automatically opened, and when the scopes are manually authorized, the authorization code is automatically retrieved.
Modified script:
import httplib2
import os
from apiclient.discovery import build
from oauth2client import client
from oauth2client import tools
from oauth2client.file import Storage
def get_service():
SCOPES = 'https://www.googleapis.com/auth/yt-analytics.readonly'
credential_path = os.path.join("./", 'tokens.json')
store = Storage(credential_path)
credentials = store.get()
if not credentials or credentials.invalid:
flow = client.flow_from_clientsecrets(CLIENT_SECRETS_FILE, SCOPES)
credentials = tools.run_flow(flow, store)
http = credentials.authorize(httplib2.Http())
return build(API_SERVICE_NAME, API_VERSION, http=http_auth)
In above script, when the script is run, when tokens.json is existing, the access token is retrieved using the refresh token in the file. By this, the authorization process using own browser is not required after 2nd run.
Patten 3:
Recently, google_auth_oauthlib is used for the authorization process. For example, you can see it at here. When this is reflected to your script, get_service() becomes as follows. Also you can see this flow at here.
Modified script:
import pickle
import os.path
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
def get_service():
SCOPES = ['https://www.googleapis.com/auth/yt-analytics.readonly']
TOKEN_FILE = 'token.pickle'
creds = None
if os.path.exists(TOKEN_FILE):
with open(TOKEN_FILE, 'rb') as token:
creds = pickle.load(token)
if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
CLIENT_SECRETS_FILE, SCOPES)
creds = flow.run_local_server()
with open(TOKEN_FILE, 'wb') as token:
pickle.dump(creds, token)
return build(API_SERVICE_NAME, API_VERSION, credentials=creds)
When the script is run, the browser is automatically opened, and when the scopes are manually authorized, the authorization code is automatically retrieved.
In above script, when the script is run, when token.pickle is existing, the access token is retrieved using the refresh token in the file. By this, the authorization process using own browser is not required after 2nd run.
Note:
In my environment, I could confirmed that when youtubeAnalytics of youtubeAnalytics = get_service() is used, execute_api_request() worked fine. If in your situation, an error occurs, please confirm whether API is enabled.
References:
Using OAuth 2.0 to Access Google APIs
Using OAuth 2.0 for Web Server Applications
Python Quickstart for Drive API
Reports: Query
If I misunderstood your question and this was not the direction you want, I apologize.