I'm writing a python lambda with an initialisation to an external service in the global section of the lambda. How do I mock this external dependency as it is initialised at the creation of my class?
data_load.py
from elasticsearch import Elasticsearch,
# Global section - executed once for the life time of the container
def get_elastic_search_connection():
try:
client = Elasticsearch(
hosts=[{'host': "host", 'port': 9191}],
use_ssl="true",
verify_certs="true",
connection_class=RequestsHttpConnection)
return client
except Exception as e:
LOGGER.error("Unable to connect to ES")
exit(1)
es_client = get_elastic_search_connection()
# handler section - executed once per function invocation
def lambda_handler(event, context):
es_client.bulk(body="body", index="index", doc_type="doc", _source=False)
In my test, as soon as the import statement imports data_load.py, it tries to initialise the global section of the lambda, which tries to make a http connection to host address. How do I create a partial mock of data_loay.py so that I can test the lambda_handler method? The es_client has to be initialised in the global section. How do I dependency inject a mock into the global section?
test_data_load.py
import unittest
import search_lambda.data_load.data_load
class TestDatLoad(unittest.TestCase):
def test_data_load(self):
print("Hello")
if __name__ == '__main__':
unittest.main()
Related
I am trying to mock secret manager. Here is the code which is written for secret manager using boto3 which I am trying to mock and test.
utils.py
import boto3
secret_id = os.environ.get("SECRETS")
client = boto3.client('secretsmanager')
response = client.get_secret_value(SecretId=secret_id)
secrets = json.loads(response['SecretString'])
S3_BUCKET_NAME = secrets["S3_BUCKET_NAME"]
SQS_QUEUE_NAME = secrets["SQS_Queue_Name"]
these variables are then used in different methods.
conftest.py
#pytest.fixture(scope='session', autouse=True)
def secret_manager_resource(aws_credentials):
"""Secret Manager mock client"""
with mock_secretsmanager():
conn = boto3.client("secretsmanager", region_name="us-east-1")
logger.info(f"Secret manager connection {conn}")
yield conn
test_file.py
#contextmanager
def secret_manager_setup(secret_manager_resource):
secret_manager_resource.create_secret(Name="test", SecretString="""{"S3_BUCKET_NAME": "test","SQS_Queue_Name": "test_queue"}""")
yield
class TestSecretManager:
def test_secret_manager(self, secret_manager_resource):
with secret_manager_setup(secret_manager_resource):
try:
result = secret_manager_resource.get_secret_value(SecretId="test")
json_result = json.loads(result['SecretString'])
assert json_result["S3_BUCKET_NAME"] == "test"
assert json_result["SQS_Queue_Name"] == "test_queue"
except Exception as err:
print("Error ---", err)
class TestClass:
def test_some_class(test_var):
from functions.something.some import something
something = someClass({}, param)
When I run pytest it directly goes inside TestClass and calls secret Manager and throws error as it is trying to connect to actual secret manager. Could someone suggest me what can be done to over come this issue?
TestClass is not mocked - so I wouldn't expect that to work. You could use Moto as a class-decorator to ensure everything inside someClass is mocked.
Note that the class-decorator creates a mock around test-methods only, so the code-under-test would have to be inside a test-method for this to work.
#mock_secretsmanager()
class TestClass:
def test_something():
from functions.something.some import something
something = someClass({}, param)
See http://docs.getmoto.org/en/latest/docs/getting_started.html#class-decorator for the documentation and more examples around this.
I have following file and i am running pytests on this file. One of the use case is to test the logger function which is global. How can we mock this in my pytest ?
import spark_fn
def run_job(params):
do_something()
logger.info("some info")
def main(params):
global logger
app_name = "test app"
spark, logger = spark_fn(app_name)
run_job(params)
if __name__ == "__main__":
params = "some params"
main(params)
===============================================================
How's something like this? You should think of it less like a global variable and more like patching what spark_fn is returning to have a "fake" logger class. Also I'm curious how you're using the spark_fn module overall with how you're importing it.
import yoursampletest
def test_monkeytest(monkeypatch):
"""
Monkey patches a fake logger into spark_fn
"""
class myFakeLogger:
def info(self, value):
return value
def mock_info(mylogstatement):
return 1, myFakeLogger()
monkeypatch.setattr(yoursampletest, 'spark_fn', mock_info)
assert yoursampletest.main(1) == 'some info'
I am trying to expose a data service as an API for a PHP application. I have written the API logic in a file called APILogic.py. The code in this looks like this
class APILogic(object):
def __init__(self):
# the initialization is done here
...
def APIfunction(self):
# the API logic is built here, it uses the class variables and methods
...
I have created another file for the API purpose. Its called API.py. The code in this file looks like this
import APILogic from APILogic
class MyFlask:
def __init__(self):
self.test = APILogic()
from flask import Flask
app = Flask(__name__)
my_flask = MyFlask()
#app.route("/Test")
def Test():
return my_flask.test.APIfunction
if __name__ == "__main__":
app.run(debug=True,port=9999)
When I run the code, I get the error
> TypeError: APIfunction() takes 1 positional argument but 3 were given
The view function did not return a valid response. The return type must be a string, dict, tuple, Response instance, or WSGI callable, but it was a method.
There are no arguments for the APIfunction though.
Please help.
The view function did not return a valid response. The return type must be a string, dict, tuple, Response instance, or WSGI callable, but it was a method.
Looks like you're returning the method, but it sounds like you want to return the result of that method:
#app.route("/Test")
def Test():
return my_flask.test.APIfunction()
View function should return valid response.
Sample API code
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "Hello World!"
if __name__ == "__main__":
app.run()
I want to understand how I can catch an HTTPException raised by flask.abort while using a test_request_context in a test for the calling method only.
# example.py
import flask
#api.route('/', methods=['POST'])
def do_stuff():
param_a = get_param()
return jsonify(a=param_a)
# check if request is json, return http error codes otherwise
def get_param():
if flask.request.is_json():
try:
data = flask.request.get_json()
a = data('param_a')
except(ValueError):
abort(400)
else:
abort(405)
# test_example.py
from app import app # where app = Flask(__name__)
from example import get_param
import flask
def test_get_param(app):
with app.test_request_context('/', data=flask.json.dumps(good_json), content_type='application/json'):
assert a == get_param()
In the get_param method above, I try to abort if the is_json() or the get_json() fail. To test this, I pass test_request_context without a content_type and, based on this blog and this answer, I tried adding a nested context manager like so:
# test_example.py
from app import app # where app = Flask(__name__)
from example import get_param
from werkzeug.exceptions import HTTPException
import flask
def test_get_param_aborts(app):
with app.test_request_context('/', data=flask.json.dumps('http://example', 'nope'), content_type=''):
with pytest.raises(HTTPException) as httperror:
get_param()
assert 405 == httperror
but I get a assert 405 == <ExceptionInfo for raises contextmanager> assertion error.
Can someone please explain this and suggest a way to test the abort in this get_param method?
Update:
Based on #tmt's answer, I modified the test. However, even though the test passes, while debugging I notice that the two assertions are never reached!
# test_example.py
from app import app # where app = Flask(__name__)
from example import get_param
from werkzeug.exceptions import HTTPException
import flask
def test_get_param_aborts(app):
with app.test_request_context('/', data=flask.json.dumps('http://example', 'nope'), content_type=''):
with pytest.raises(HTTPException) as httperror:
get_param() # <-- this line is reached
assert 405 == httperror.value.code
assert 1 ==2
httperror is an instance of ExceptionInfo which is pytest's own class that describes the exception. Once it happens, httperror would also contain value property which would be the instance of the HTTPException itself. If my memory is correct HTTPException contains code property that equals to the HTTP status code so you can use it to do the assertion:
# test_example.py
from app import app
from example import get_param
from werkzeug.exceptions import HTTPException
import flask
def test_get_param_aborts(app):
with app.test_request_context('/', data=flask.json.dumps(), content_type=''):
with pytest.raises(HTTPException) as httperror:
get_param()
assert 405 == httperror.value.code
Notes:
get_param() needs to be called within pytest.raises() context manager.
The assertion needs to be done outside of the context manager because once an exception is raised the context ends.
I don't know if pytest.raise is your typo or if it really existed in older versions of pytest. AFAIK it should be pytest.raises.
When testing my Pyramid application using WebTest, I have not been able to create/use a separate Session in my tests without getting warnings about a scoped session already being present.
Here is the main() function of the Pyramid application, which is where the database is configured.
# __init__.py of Pyramid application
from pyramid_sqlalchemy import init_sqlalchemy
from sqlalchemy import create_engine
def main(global_config, **settings):
...
db_url = 'some-url'
engine = create_engine(db_url)
init_sqlalchemy(engine) # Warning thrown here.
Here is the test code.
# test.py (Functional tests)
import transaction
from unittest import TestCase
from pyramid.paster import get_appsettings
from pyramid_sqlalchemy import init_sqlalchemy, Session
from sqlalchemy import create_engine
from webtest import TestApp
from app import main
from app.models.users import User
class BaseTestCase(TestCase):
def base_set_up(self):
# Create app using WebTest
settings = get_appsettings('test.ini', name='main')
app = main({}, **settings)
self.test_app = TestApp(app)
# Create session for tests.
db_url = 'same-url-as-above'
engine = create_engine(db_url)
init_sqlalchemy(engine)
# Note: I've tried both using pyramid_sqlalchemy approach here and
# creating a "plain, old" SQLAlchemy session here using sessionmaker.
def base_tear_down(self):
Session.remove()
class MyTests(BaseTestCase):
def setUp(self):
self.base_set_up()
with transaction.manager:
self.user = User('user#email.com', 'John', 'Smith')
Session.add(self.user)
Session.flush()
Session.expunge_all()
...
def tearDown(self):
self.base_tear_down()
def test_1(self):
# This is a typical workflow on my tests.
response = self.test_app.patch_json('/users/{0}'.format(self.user.id), {'email': 'new.email#email.com')
self.assertEqual(response.status_code, 200)
user = Session.query(User).filter_by(id=self.user.id).first()
self.assertEqual(user.email, 'new.email#email.com')
...
def test_8(self):
...
Running the tests gives me 8 passed, 7 warnings, where every test except the first one gives the following warning:
From Pyramid application: __init__.py -> main -> init_sqlalchemy(engine):
sqlalchemy.exc.SAWarning: At least one scoped session is already present. configure() can not affect sessions that have already been created.
If this is of any use, I believe I am seeing the same issue here, except I am using pyramid_sqlalchemy rather than creating my own DBSession.
https://github.com/Pylons/webtest/issues/5
Answering my own question: I'm not sure if this is the best approach, but one that worked for me.
Instead of trying to create a separate session within my tests, I am instead using the pyramid_sqlalchemy Session factory, which is configured in the application. As far as I can tell, calls to Session in both test and application code return the same registered scoped_session.
My original intent with creating a separate session for my tests was to confirm that records were being written to the database, and not just updated in the active SQLAlchemy session. With this new approach, I've managed to avoid these "caching" issues by issuing Session.expire_all() at points in the tests where I transition between test transactions and application transactions.
# test.py (Functional tests)
import transaction
from unittest import TestCase
from pyramid.paster import get_appsettings
from pyramid_sqlalchemy import Session
from webtest import TestApp
from app import main
from app.models.users import User
class BaseTestCase(TestCase):
def base_set_up(self):
# Create app using WebTest
settings = get_appsettings('test.ini', name='main')
app = main({}, **settings)
self.test_app = TestApp(app)
# Don't set up an additional session in the tests. Instead import
# and use pyramid_sqlalchemy.Session, which is set up in the application.
def base_tear_down(self):
Session.remove()
class MyTests(BaseTestCase):
def setUp(self):
self.base_set_up()
with transaction.manager:
self.user = User('user#email.com', 'John', 'Smith')
Session.add(self.user)
Session.flush()
Session.expunge_all()
Session.expire_all() # "Reset" your session.
def tearDown(self):
self.base_tear_down()