I have nestjs application, For unit testing, I need to test getting the files data from azure storage file share, for that I'm using #azure/storage-file-share. But I dont want access the actual storage account because, in our github pipeline runner it is not accessible. Is there a way to test storage file share on local/cicd pipeline without trying to access the actual service? P.S. I'm using jest to write unit tests.
I want to do the same thing with azure service bus.
A lot of this will depend on how you have set up your app, but your best bet is mocking Storage and Service Bus. See the docs for mocking a full npm module, but once you've done that, any calls to the Azure services won't actually happen, they will just do nothing and you can still test that the calls were made.
However, if your tests rely on data they pulled data from storage, I would make sure you have separated the data layer into a separate interface. For example:
export const loadData = (key) => {
// use Storage library to get the file
// ...
return data;
}
export const saveData = (key) => {
// use Storage library to save the file
// ...
return success;
}
If you've hidden the file implementation behind an interface like this, you can now just mock loadData():
import * from './myDataInterface';
// ... more code ...
// 'my data' could be a string, object, whatever you need the loaded file to be to set up your test
const mock = jest.spyOn(myDataInterface, 'loadData').mockReturnValue('my data');
And now when the code that you are testing calls loadData() it will be running the mock and returning what you expected it to, not calling the storage module.
Much more about mocking in the jest docs, I wouldn't presume that this is 100% going to fit your needs but hopefully it is a good start.
Related
I have a Node.js application and it already has Unit Tests and is using the Mocha framework for the same. It is checking the functions individually. These tests are integrated into the CI/CD pipeline in Bamboo and so if there is an error, it will stop the build job and alert the user who has pushed the change.
Now I have a requirement that I need to validate a JSON file, which is available on one of the S3 buckets. It downloads the file once the Node.js application is started in the local environment. I have unit tests to check whether the downloading functionality is working or not and it is working fine. Now for the validation purpose, I am a little confused about whether I need to add it as a unit test or an integration test. I am new to QA and I would like to do it in the right way. As of now, there are no integration tests are in place(No tests are checking the API endpoints). It will be helpful if someone can point me in the right direction. Also, it will be helpful if someone can suggest the framework I need to use with Node.js for writing integration tests.
I have the following code that is used for testing the download functionality.
it(`Download file from S3`, (done) => {
s3Service.getJSONFile('','',Date.now()).then(data => {
setTimeout(() => {
assert.equal(data, "JSON File Download Success");
done();
}, 1000);
}).catch(function(error){
console.log("Error in getJSONFileFromS3: "+ JSON.stringify(error))
})
});
I have a function validateJSON for validating the JSON file and its contents. Not sure whether I need to call this function from a Unit Test so that it will return true or false. But I think in the case of Unit Test it will check whether the validation function is working or not and not the validity of the file. What I need is for my tests should succeed if the JSON file is valid and fail if it is not so that the build will be stopped. By the way, I don't have an API endpoint for the JSON validation
It will be helpful if someone can show me an example of how these types of scenarios should be addressed in testing.
I have an Azure Function that will call an external API via HttpClient. The external API returns a JSON response. I want to save the response directly to an ADLS File.
My simplistic code is:
public async Task UploadFileBulk(Stream contentToUpload)
{
await this._theClient.FileClient.UploadAsync(contentToUpload);
}
The this._theClient is a simple wrapper class around the various Azure Data Lake classes such as DataLakeServiceClient, DataLakeFileSystemClient, DataLakeDirectoryClient, DataLakeFileClient.
I'm happy this wrapper calls works as I expect, I spin one up, set the service, filesystem, directory and then a filename to create. I've used this wrapper class to create directories etc. so it works as I expect.
I am calling the above method as follows:
await dlw.UploadFileBulk(await this._httpClient.GetStreamAsync("<endpoint>"));
I see the file getting created in the Lake directory with the name I want, however if I then download the file using Sorage Explorer and then try to open it in say VS Code it's not in a recognisable format (I can "force" code to open it but it looks like binary format to me).
If I sniff the traffic with fiddler I can see the content from the external API is JSON, content-type is application/json and the body shows in fiddler as JSON.
If I look at the calls to the ADLS endpoint I can see a PUT call followed by two PATCH calls.
The first PATCH call looks like it is the one sending the content, it has a content-header of application/octet-stream and the request body is the "binary looking content".
I am using HttpClient.GetStreamAsync as I don't want my Function to have to load the entire API payload into memory (some of the external API endpoints return very large files over 100mb). I am thinking I can "stream the response from the external API straight into ADLS".
Is there a way to change how the ADLS FileClient.UploadAsync(Stream stream) method works so I can tell it to upload the file as a JSON file with a content type of application/json?
EDIT:
So turns out the External API was sendng back zipped content and so once I added the following extra AutomaticDecompression code to my functions startup I got the files uploaded to ADLS as expected.
public override void Configure(IFunctionsHostBuilder builder)
{
builder.Services.AddHttpClient("default", client =>
{
client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate");
}).ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
});
}
#Gaurav Mantri has given me some pointers on if the pattern of "streaming from an output to an input" is actually correct, I will research this further.
Regarding the issue, please refer to the following code
var uploadOptions = new DataLakeFileUploadOptions();
uploadOptions.HttpHeaders = new PathHttpHeaders();
uploadOptions.HttpHeaders.ContentType ="application/json";
await fileClient.UploadAsync(stream, uploadOptions);
So I am using firebase functions, as my API request with some functions doing some work,
Right now I have 1 'index.ts' file and all of my functions inside (14 functions) there with this signature:
export const $function1 = functions.region(#region).runWith({ memory: '512MB' }).https.onRequest
export const $function2 = functions.region(#region).https.onCall(async (data, context) => {
Now my question is about organize this 'index.ts',
Some of my https requests verify token and check for valid body request,
What is the best way to organize this 'index.ts' file?
How exactly I am exporting it from files in this folder without initialize the admin sdk every time?
I am trying to make this folder organize as I can as in the future I need to add more and more functions..
Thanks for the help!
This is really a personal preference but the recommended way to organize functions by Firebase documentation is to write all your functions in their own files, export them then initialize them all in one index.ts file[1]. You can also group your functions depending on their purpose like auth or logging ...etc then export them to one index.ts file. For more info, check the documentation I have shared but as I said, you can always organize them in your own way.
[1]https://firebase.google.com/docs/functions/organize-functions
I have just started my first loopback project and chosen loopback4 version for the application. Its purely a server application which will interact with databases (Redis and mongodb) and will call external API services due to micro-service architecture.
Now, I have 3 datasources in my application i.e. mongodb, Redis, and REST based datasource to call external services. I am facing 2 problems in going forward.
1. Environment specific configurations of Datasources: I need to maintain configuration for all three datasources according to the NODE_ENV environment variable. For lb3 i found this solution,
https://loopback.io/doc/en/lb3/Environment-specific-configuration.html#data-source-configuration
which does not work in lb4. One solution is to add configuration files having names mongodb.staging.json and mongodb.production.json and same for redis and rest datasources in directory src/datasources, and load this config according to NODE_ENV variable using if condition and pass it to the constructor of datasource. It works but it does not seem nice, as it should be application's responsibility to do this.
Can somebody suggest me lb3 equivalent solution for the above?
2. Calling External APIs via datasource: in lb4, To call external services its recommended to have a separate REST based datasource and its service to call it via controller. Now, In REST datasource config, one has to define a template of all the API calls which will happen to the external service https://loopback.io/doc/en/lb4/REST-connector.html#defining-a-custom-method-using-a-template.
As my application calls external service heavily with relatively large number of request parameters. It becomes really messy to declare each API call with its request params and to maintain this in the datasource config which will be environment specific.
Can somebody tell me a more robust and cleaner alternative of the above problem?
Thanks in advance!!
Using environment variables in datasource configs
The datasource config is simply a JSON file that's imported in into *.datasource.ts. Hence, you can replace that JSON file with a Typescript file and import it accordingly. LoopBack 4 does not provide any custom variable substitution mechanism. Instead, it is recommended to use process.env.
Recent CLI versions replace the JSON config in favour of using a single Typescript file:
import {inject} from '#loopback/core';
import {juggler} from '#loopback/repository';
const config = {
name: 'db',
connector: 'memory',
};
export class DbDataSource extends juggler.DataSource {
static dataSourceName = 'db';
static readonly defaultConfig = config;
constructor(
#inject('datasources.config.db', {optional: true})
dsConfig: object = config,
) {
super(dsConfig);
}
}
The dependency injection in the constructor allows you to override the config programmatically via the IoC container of the application.
Further reading
https://loopback.io/doc/en/lb4/DataSources.html
Calling external APIs without REST connector
The REST connector enforces a well-defined interface for querying external APIs so as to be able to do validation before sending out the request.
If this is not favourable, it is possible to create a new Service as a wrapper to the HTTP queries. From there, you can expose your own functions to handle requests to an external API. As Services do not need to follow a rigid structure, it is possible to customize it to your use-case.
It is also possible to create a new request directly inside the controller using either built-in or external libraries.
Overall, there isn't a 100% right or wrong way of doing certain things in LoopBack 4. Hence why the framework provides numerous ways to tackle the same issue.
I am doing some learnings with Terraform and sentinel.
I cant get some of the basic functionality working.
I have a policy here:
import "tfconfig"
default_foo = rule { tfconfig.variables.foo.default is "bar" }
default_number = rule { tfconfig.variables.number.default is 42 }
main = rule { default_foo and default_number }
and a variables file here:
variable "foo" {
default = "bar"
}
variable "number" {
default = 42
}
But when I run:
sentinel apply policy.sentinel
I get the following error:
policy.sentinel:1:1: Import "tfconfig" is not available.
Any ideas as I have been looking for a solution for a number of hours now.
thanks
In order to use the Terraform-specific imports in the Sentinel SDK, you need to use mock data to produce a data structure to test against.
When you run Terraform via Terraform Cloud, a successful plan will produce a Sentinel mocks file that contains the same data that Terraform Cloud would itself use when evaluating policies against that plan, and so you can check that mock data into your repository as part of your test suite for your policies.
You can use speculative plans (run terraform plan on the command line with the remote backend enabled) to create mock data for intentionally-invalid configurations that you want to test your policy against, without having to push those invalid configurations into your version control system.
You can use sentinel test against test cases whose JSON definitions include a mock object referring to those mock files, and then the policies evaluated by those test cases will be able to import tfconfig, tfplan and tfstate and get an equivalent result to if the policies were run against the original plan in Terraform Cloud.
I had my mock data and everything and I was still getting this error.
Runtime error while running the policy:
ensure-policy.sentinel:3:1: Import "tfplan" is not available
I realized I had forgotten the 'sentinel.json' file at the root of my policies dir, which tells sentinel where to look for the mock data
https://www.terraform.io/docs/cloud/sentinel/mock.html
here's how they recommend a default dir setup, but you still need that .json file to tell sentinel where to look
and here's what the sentinel.json file should look like