I have search about this issue and couldn't find anything that would help me.
EDITED
The main idea is to, each time the lambda function is triggered by cloudwatch (everyday), choose a subsequent line from the text file that I get from a s3 bucket, that line will be attached to an e-mail.
The next time the lambda is triggered, the same will happen, but with the next line in the text file, and so on.
I have more or less an idea, using a for loop, my problem is how to, each time the function is triggered, select the next line in the text file.
If you need to read lines from your file in sequence, one lambda execution a day is one line, then you have to keep track of those lines. If its only once a day, you could use SSM Parameter Store for that. Each time your lambda executes, it would query SSM Parameter Store the the line number which was read previously.
Similarly, after successful dispatch of a line, the Lambda function would update the parameter in SSM Parameter Store.
The exact details depend on how big the file is, as this process can get progressively slow with time.
Related
I have a Dash application that queries an API, based on a user search query, performs some calculations on the response, then displays the final results to the user on a Dash app. In order to provide a quick response to the user, I am trying to set up a quick result callback and a full result long_callback.
The quick result will grab limited results from the API and display results to the user within 10-15 seconds, while the full results will run in the background, collecting all results (which can take up to 2 minutes), then updates the page with the full results when they are available.
I am curious what the best way to perform this action is, as I have run into forking issues with my current attempt.
My current attempt: Using the diskcache.Cache() as the DiskcacheLongCallbackManager and a database txt file to store availability of results.
I have a database txt file that stores a dictionary, with the keys being the search query and the fields being quick_results: bool, full_results: bool, file_path: str, timestamp: dt (as str).
When a search query is entered and submit is pressed, a callback loads the database file as a variable and then checks the dictionary keys for the presence of this search query.
If it finds the query in the keys of the database, it loads the saved feather file from the provided file_path and returns it to the dash app for generation of the page content.
If it does not find the query in the database keys, it requests limited data from the API, runs calculations, saves the DataFrame as a feather file on disk, then creates an entry in the database with the search query(as the key), the file path of the saved feather file, the current timestamp, and sets the quick_results value to True.
It then loads this feather file from the file_path created and returns it to the dash app for generation of the page content.
A long_callback is triggered at the same time as the above callback, with a 20 second sleep to prevent overlap with the quick search. This callback also loads the database file as a variable and checks if the query is present in the database keys.
If found, it then checks if the full results value is True and if the timestamp is more than 0 days old.
If the full results are unavailable or are more than 0 days old, the long_callback requests full results from the API, performs the same calculations, then updates the already existing search query in the database, making the full_results True and the timestamp the time of completion for the full search.
It then loads the feather file from the file_path and returns it to the dash app for generation of the page content.
If the results are available and less then 1 day old, the long callback simply loads the feather file from the provided file_path and returns it to the dash app for generation of the page content.
The problem I am currently facing is that I am getting a weird forking error on the long callback on only one of the conditions for a full search. I currently have the long_callback setup to perform a full search only if the full results flag is False or the results are more than 0 days old. When the full_results flag is False, the callback runs as expected, updates the database and returns the full results. However, when the results are available but more than 0 days old, the callback hits a forking error and is unable to complete.
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec(). Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
I am at a loss as to why the function would run without error on one of the conditions, but then have a forking error on the other condition. The process that runs after both conditions is exactly the same.
By using print statements, I have noticed that this forking error triggers when the function tries to call the requests.get() function on the API.
If this issue is related to how I have setup the background process functionality, I would greatly appreciate some suggestions or assitance on how to do this properly, where I will not face this forking error.
If there is any information I have left out that will be helpful, please let me know and I will try to provide it.
Thank you for any help you can provide.
I am trying to understand the code behind this solution. This function takes a list of file name and sort by extension (filename.extension is the format) and puts full file name with extension only but not a name (.config for example) first. I am having a hard time understanding the 3 arguments within the lambda function. What does each one of them do?
def sort_by_ext(files):
return sorted(files,key=lambda x:(bool(i:=x.rfind('.')),x[i+1:],x[:i]))
Let's break it down:
bool(i := x.rfind('.'))
This will assign to i the last position that a period is found in the string x. It will also return True if it has a filename (basically, if the last period is not at the beginning), and False if it doesn't (basically, if the last period is at the beginning).
x[i+1:]
This will return the file extension (basically, everything after the last period).
x[:i]
This will return the file name without the file extension (basically, everything before the last period).
To wrap it all up, the code sorts by the following (in the order it appears below):
Whether there is a filename
The file extension
The filename minus the file extension
I have a node.js app, which opens a stream:
outputStream = fs.createWriteStream("output.txt");
I then, asynchronously, add text to the file:
outputStream.write( outputTxt, "utf8" );
This code is being run inside a loop, so it happens hundreds of times. However, the loop is asynchronous, so it sometimes pauses, and I can edit the output.txt file in an external editor in the meantime, and (for example) add a few chars in the beginning.
However, when I do that, the next time the outputStream.write is executed, it overwrites the last few chars previously added (the same number of chars that I added externally).
Is there some way to prevent this? Some way to tell the writeStream find the end of the file and then add the text?
I'm trying to Create a file and append all the content being calculated into that file, but when I run the script the very last iteration is written inside the file and nothing else.
My code is on pastebin, it's too long, and I feel like you would have to see exactly how the iteration is happening.
Try to summarize it, Go through an array of model numbers, if the model number matches call the function that calculates that MAC_ADDRESS, when done calculating store all the content inside a the file.
I have tried two possible routes and both have failed, giving the same result. There is no error in the code (it runs) but it just doesn't store the content into the file properly there should be 97 different APs and it's storing only 1.
The difference between the first and second attempt,
1 attempt) I open/create file in the beginning of the script and close at the very end.
2 attempt) I open/create file and close per-iteration.
First Attempt:
https://pastebin.com/jCpLGMCK
#Beginning of code
File = open("All_Possibilities.txt", "a+")
#End of code
File.close()
Second Attempt:
https://pastebin.com/cVrXQaAT
#Per function
File = open("All_Possibilities.txt", "a+")
#per function
File.close()
If I'm not suppose to reference other websites, please let me know and I'll just paste the code in his post.
Rather than close(), please use with:
with open('All_Possibilities.txt', 'a') as file_out:
file_out.write('some text\n')
The documentation explains that you don't need + to append writes to a file.
You may want to add some debugging console print() statements, or use a debugger like pdb, to verify that the write() statement actually ran, and that the variable you were writing actually contained the text you thought it did.
You have several loops that could be a one-liner using readlines().
Please do this:
$ pip install flake8
$ flake8 *.py
That is, please run the flake8 lint utility against your source code,
and follow the advice that it offers you.
In particular, it would be much better to name your identifier file than to name it File.
The initial capital letter means something to humans reading your code -- it is
used when naming classes, rather than local variables. Good luck!
1.Validate Line Function
Does this script fire at field level of Line Items OR fire at the record level?
2.Scheduled Script
How to test Performance of a Script? Would debug help or are there any other logs to check on bottlenecks (delays)?
Validate Line fires at the line level, i.e. not until you click "Done" or moving to the next line. Your event handler function can return false to stop the line addition/update from happening, or true to continue normally.
As far as I know, currently the debugger can only be used on User Event scripts. You could do some simple time measurements and log timings throughout your script. Your most likely points of bottlenecks will be anything that goes back to the database (e.g. creating/loading/submitting records, complex searches, etc).