I have an HTTP triggered Consumption plan Azure Function that I want to keep warm by POSTing an empty payload to it regularly.
I am doing this with a Scheduled Function with this configuration:
__init__.py
import os
import datetime
import logging
import azure.functions as func
import urllib.parse, urllib.request, urllib.error
def main(mytimer: func.TimerRequest) -> None:
try:
url = f"https://FUNCTIONNAME.azurewebsites.net/api/predictor?code={os.environ['CODE']}"
request = urllib.request.Request(url, {})
response = urllib.request.urlopen(request)
except urllib.error.HTTPError as e:
message = e.read().decode()
if message == "expected outcome":
pass
else:
logging.info(f"Error: {message}")
function.json
{
"scriptFile": "__init__.py",
"bindings": [
{
"name": "mytimer",
"type": "timerTrigger",
"direction": "in",
"schedule": "0 */9 5-17 * * 1-5"
}
]
}
When I inspect my logs they are filled with HTML. Here is a snippet of the HTML:
...
<h1>Server Error</h1>
...
<h2>502 - Web server received an invalid response while acting as a gateway or proxy server.</h2>
<h3>There is a problem with the page you are looking for, and it cannot be displayed. When the Web server (while acting as a gateway or proxy) contacted the upstream content server, it received an invalid response from the content server.</h3>
Running the logic of __init__.py locally works fine. What might be wrong here?
Hmm... That is weird. Looks like the response wasn't able to route to the correct instance I guess.
BTW, I believe you could simply have the time triggered function in the same function app as the one you want to keep warm. This function really doesn't have to do anything too.
Also, you might want to take a look at Azure Functions Premium which supports having pre-warmed instances. Note that this is still in preview.
Related
I've created an Azure Function that fetches data from an External API and sends that data to my cosomos container using a timer trigger. This function is not sending any data to the cosmos db and I can't figure out why.
I've removed some lines of code which contained sensitive information, but I have tested the code locally and it is fetching the data and sending to cosmos if I just run the file.
here is my function:
import datetime
import logging
import azure.functions as func
import time
import pandas as pd
import requests
import json
import uuid
from azure.cosmos.aio import CosmosClient
import asyncio
def get_data():
url = 'externalAPIUrl'
session = requests.Session()
request = session.get(url, headers=headers)
cookies = dict(request.cookies)
response = session.get(url, headers=headers, cookies=cookies).json()
rawData = pd.DataFrame(response)
rawop = pd.DataFrame(rawData['filtered']['data'])
rawop = rawop.set_index('paramter')
processed_data = []
for i in rawop.index:
#creating json to send
processed_data.append(processed_ce)
processed_data.append(processed_pe)
logging.info("Data processed successfully")
return processed_data
async def manage_cosmos(processed_data):
print(len(processed_data))
cosmosdb_endpoint = 'endpointURL'
cosmos_key = 'key'
DATABASE_NAME = 'dbname'
CONTAINER_NAME = 'containername'
async with CosmosClient(cosmosdb_endpoint, cosmos_key) as client:
database = client.get_database_client(DATABASE_NAME)
container = database.get_container_client(CONTAINER_NAME)
# send all the objects in the processed_Data list to cosmos asyncronously
tasks = []
for i in processed_data[:40]:
tasks.append(container.create_item(i))
await asyncio.gather(*tasks)
tasks = []
for i in processed_data[40:90]:
tasks.append(container.create_item(i))
await asyncio.gather(*tasks)
tasks = []
for i in processed_data[90:]:
tasks.append(container.create_item(i))
await asyncio.gather(*tasks)
logging.info("Data sent to cosmos successfully")
def main(mytimer: func.TimerRequest) -> None:
if mytimer.past_due:
logging.info('The timer is past due!')
try:
processed_data = get_data()
asyncio.run(manage_cosmos(processed_data))
except Exception as e:
logging.error("error in function", e)
here is my function.json
{
"scriptFile": "__init__.py",
"bindings": [
{
"name": "mytimer",
"type": "timerTrigger",
"direction": "in",
"schedule": "0 */2 * * * *"
}
]
}
this is my requirements.txt
# DO NOT include azure-functions-worker in this file
# The Python Worker is managed by Azure Functions platform
# Manually managing azure-functions-worker may cause unexpected issues
azure-functions
pandas
asyncio
aiohttp
requests
uuid
azure-cosmos
Can somebody tell me why it is not sending data to cosmos?
I deployed the above function and no data was added to my cosmos container. I ran the same code without the azure-function part locally and I was able to send new items to my cosmos container.
I am most probably making a mistake in creating the function so if anybody can point that out.
I've setup a Python script that will take certain bigquery tables from one dataset, clean them with a SQL query, and add the cleaned tables to a new dataset. This script works correctly. I want to set this up as a cloud function that triggers at midnight every day.
I've also used cloud scheduler to send a message to a pubsub topic at midnight every day. I've verified that this works correctly. I am new to pubsub but I followed the tutorial in the documentation and managed to setup a test cloud function that prints out hello world when it gets a push notification from pubsub.
However, my issue is that when I try to combine the two and automate my script - I get a log message that the execution crashed:
Function execution took 1119 ms, finished with status: 'crash'
To help you understand what I'm doing, here is the code in my main.py:
# Global libraries
import base64
# Local libraries
from scripts.one_minute_tables import helper
def one_minute_tables(event, context):
# Log out the message that triggered the function
print("""This Function was triggered by messageId {} published at {}
""".format(context.event_id, context.timestamp))
# Get the message from the event data
name = base64.b64decode(event['data']).decode('utf-8')
# If it's the message for the daily midnight schedule, execute function
if name == 'midnight':
helper.format_tables('raw_data','table1')
else:
pass
For the sake of convenience, this is a simplified version of my python script:
# Global libraries
from google.cloud import bigquery
import os
# Login to bigquery by providing credentials
credential_path = 'secret.json'
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = credential_path
def format_tables(dataset, list_of_tables):
# Initialize the client
client = bigquery.Client()
# Loop through the list of tables
for table in list_of_tables:
# Create the query object
script = f"""
SELECT *
FROM {dataset}.{table}
"""
# Call the API
query = client.query(script)
# Wait for job to finish
results = query.result()
# Print
print('Data cleaned and updated in table: {}.{}'.format(dataset, table))
This is my folder structure:
And my requirements.txt file has only one entry in it: google-cloud-bigquery==1.24.0
I'd appreciate your help in figuring out what I need to fix to run this script with the pubsub trigger without getting a log message that says the execution crashed.
EDIT: Based on the comments, this is the log of the function crash
{
"textPayload": "Function execution took 1078 ms, finished with status: 'crash'",
"insertId": "000000-689fdf20-aee2-4900-b5a1-91c34d7c1448",
"resource": {
"type": "cloud_function",
"labels": {
"function_name": "one_minute_tables",
"region": "us-central1",
"project_id": "PROJECT_ID"
}
},
"timestamp": "2020-05-15T16:53:53.672758031Z",
"severity": "DEBUG",
"labels": {
"execution_id": "x883cqs07f2w"
},
"logName": "projects/PROJECT_ID/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
"trace": "projects/PROJECT_ID/traces/f391b48a469cbbaeccad5d04b4a704a0",
"receiveTimestamp": "2020-05-15T16:53:53.871051291Z"
}
The problem comes from the list_of_tables attributes. You call your function like this
if name == 'midnight':
helper.format_tables('raw_data','table1')
And you iterate on your 'table1' parameter
Perform this, it should work
if name == 'midnight':
helper.format_tables('raw_data',['table1'])
When using the Python client API for the Google Cloud Scheduler I always get the above error message for some reason. I also tried to start the parent path without the slash but got the same result.
Any hint is much appreciated!
import os
from google.cloud import scheduler_v1
def gcloudscheduler(data, context):
current_folder = os.path.dirname(os.path.abspath(__file__))
abs_auth_path = os.path.join(current_folder, 'auth.json')
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = abs_auth_path
response = scheduler_v1.CloudSchedulerClient().create_job(data["parent"], data["job"])
print(response)
I used following parameter:
{"job": {
"pubsub_target": {
"topic_name": "trade-tests",
"attributes": {
"attrKey": "attrValue"
}
},
"schedule": "* * * * *"
},
"parent": "/projects/my-project-id/locations/europe-west1"
}
The problem was actually not the parent parameter but the incorrect format of the topic-name. It should have been projects/my-project-id/topics/trade-tests. Even though the error message says it should with a slash. But it is in line with the API doc here and here.
The problem was just that the error message didn't say which resource name the error was about.
I have a flask app deployed to Heroku and would like to receive text from Chatfuel (bot building platform) and send back texts in return.
Now, what I did is to use my heroku app as a web-hook, so that Chatfuel can make a simple GET or POST query to my API. The problem is that I have no experience with Flask or APIs, so I am not sure about how my app can receive information (in json format) and send it back to chatfuel.
This is what I wrote so far:
import os
import sys
import json
import requests
from flask import Flask, jsonify, render_template, request
app = Flask(__name__)
#app.route('/', methods=['GET'])
def verify():
# when the endpoint is registered as a webhook, it must echo back
# the 'hub.challenge' value it receives in the query arguments
if request.args.get("hub.mode") == "subscribe" and request.args.get("hub.challenge"):
if not request.args.get("hub.verify_token") == os.environ["VERIFY_TOKEN"]:
return "Verification token mismatch", 403
return request.args["hub.challenge"], 200
return "Hello world", 200
#app.route("/json", methods=['GET','POST'])
def json():
url = "chatfuel_api"
data = json.load(urllib2.urlopen(url))
if request.json:
mydata = request.json
return "Thanks",200
else:
return "no json received"
#app.route('/hello', methods = ['GET','POST'])
def api_echo():
if request.method == 'GET':
return "ECHO: GET\n",200
if __name__ == '__main__':
app.run(debug=True)
The verify() function works, as I see an 'Hello world' message if I run the app locally. However, both json() and api_echo() don't work, and when my server receives a get or post request from chatfuel, it returns a 404 error.
As you can see, I really have a lot of confusion, and your help would be really invaluable.
Thanks
You need to make sure you have registered the proper webhook url with Chatfuel. For the code you currently have, to hit the json endpoint the url would be https://www.your_server.com/json
The verify route looks like the hub challenge FB sends, so you would have to register the root of your site (that is, with your current code) with FB to hit the verify function. That url would look like this https://www.your_site.com/
Hy. I try to write test for webHander:
import pytest
import tornado
from tornado.testing import AsyncTestCase
from tornado.httpclient import AsyncHTTPClient
from tornado.web import Application, RequestHandler
import urllib.parse
class TestRESTAuthHandler(AsyncTestCase):
#tornado.testing.gen_test
def test_http_fetch_login(self):
data = urllib.parse.urlencode(dict(username='admin', password='123456'))
client = AsyncHTTPClient(self.io_loop)
response = yield client.fetch("http://localhost:8080//#/login", method="POST", body=data)
# Test contents of response
self.assertIn("Automation web console", response.body)
Received error when running test:
raise TimeoutError('Operation timed out after %s seconds' % timeout)
tornado.ioloop.TimeoutError: Operation timed out after 5 seconds
Set ASYNC_TEST_TIMEOUT environment variable.
Runs the IOLoop until stop is called or timeout has passed.
In the event of a timeout, an exception will be thrown. The default timeout is 5 seconds; it may be overridden with a timeout keyword argument or globally with the ASYNC_TEST_TIMEOUT environment variable. -- from http://www.tornadoweb.org/en/stable/testing.html#tornado.testing.AsyncTestCase.wait
You need to use AsyncHTTPTestCase, not just AsyncTestCase. A nice example is in Tornado's self-tests:
https://github.com/tornadoweb/tornado/blob/d7d9c467cda38f4c9352172ba7411edc29a85196/tornado/test/httpclient_test.py#L130-L130
You need to implement get_app to return an application with the RequestHandler you've written. Then, do something like:
class TestRESTAuthHandler(AsyncHTTPTestCase):
def get_app(self):
# implement this
pass
def test_http_fetch_login(self):
data = urllib.parse.urlencode(dict(username='admin', password='123456'))
response = self.fetch("http://localhost:8080//#/login", method="POST", body=data)
# Test contents of response
self.assertIn("Automation web console", response.body)
AsyncHTTPTestCase provides convenient features so you don't need to write coroutines with "gen.coroutine" and "yield".
Also, I notice you're fetching a url with a fragment after "#", please note that in real life web browsers do not include the fragment when they send the URL to the server. So your server would see the URL only as "//", not "//#/login".