Python How to send multiple files - python-3.x

How to make bot upload multiple image with content description. i can add multiple lines it works, but bot slows posting when it has more than 5 await bot.send line. i need to add few images so how to do it if possible in same line.
#bot.command(pass_context=True)
async def ping(ctx):
await bot.send_file(ctx.message.channel, "Image1.png", content="Image1")

You want to run multiple asynchronous tasks in one await.
You should use asyncio.wait:
import asyncio
#bot.command(pass_context=True)
async def ping(ctx):
files = ... # Set the 5 files (or more ?) you want to upload here
await asyncio.wait([bot.send_file(ctx.message.channel, f['filename'], content=f['content'] for f in files)])
(See Combine awaitables like Promise.all)

Related

Python Playwright Download only certain files from a page

I'm attempting to download files from a page that's constructed almost entirely in JS. Here's the setup of the situation and what I've managed to accomplish.
The page itself takes upward of 5 minutes to load. Once loaded, there are 45,135 links (JS buttons). I need a subset of 377 of those. Then, one at a time (or using ASYNC), click those buttons to initiate the download, rename the download, and save it to a place that will keep the download even after the code has completed.
Here's the code I have and what it manages to do:
import asyncio
from playwright.async_api import async_playwright
from pathlib import Path
path = Path().home() / 'Downloads'
timeout = 300000 # 5 minute timeout
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch()
context = await browser.new_context()
page = await context.new_page()
await page.goto("https://my-fun-page.com", timeout=timeout)
await page.wait_for_selector('ul.ant-list-items', timeout=timeout) # completely load the page
downloads = page.locator("button", has=page.locator("span", has_text="_Insurer_")) # this is the list of 377 buttons I care about
# texts = await downloads.all_text_contents() # just making sure I got what I thought I got
count = await downloads.count() # count = 377.
# Starting here is where I can't follow the API
for i in range(count):
print(f"Starting download {i}")
await downloads.nth(i).click(timeout=0)
page.on("download", lambda download: download.save_as(path / download.suggested_filename))
print("\tDownload acquired...")
await browser.close()
asyncio.run(main())
UPDATE: 2022/07/15 15:45 CST - Updated code above to reflect something that's closer to working than previously but still not doing what I'm asking.
The code above is actually iterating over the locator object elements and performing the downloads. However, the page.on("download") step isn't working. The files are not showing up in my Downloads folder after the task is completed. Thoughts on where I'm missing the mark?
Python 3.10.5
Current public version of playwright
First of all, download.save_as returns a coroutine which you need
to await. Since there is no such thing as an "aysnc lambda", and
that coroutines can only be awaited inside async functions, you
cannot use lambda here. You need to create a separate async function,
and await download.save_as.
Secondly, you do not need to repeatedly call page.on. After registering it once, the callable will be called automatically for all "download" events.
Thirdly, you need to call page.on before the download actually happens (or before the event fires, in general). It's often best to shift these calls right after you create the page using .new_page().
A Better Solution
These were the obvious mistakes in your approach, fixing them should make it work. However, since you know exactly when the downloads are going to take place (after you click downloads.nth(i)), I would suggest using expect_download instead. This will make sure that the file is completely downloaded before your main program continues (callables registered with events using page.on are not awaited). Your code will somewhat become like this:
for i in range(count):
print(f"Starting download {i}")
async with page.expect_download() as download_handler:
await downloads.nth(i).click(timeout=0)
download = await download_handler.value
await download.save_as(path + f'\\{download.suggested_filename}')
print("\tDownload acquired...")

Download multiple GCS files in parallel (into memory) using Python

I have a storage bucket with a lot of large files (500mb each). At times I need to load multiple files, referenced by name. I have been using the blob.download_as_string() function to download the files one-by-one, but it's extremely slow so I would like to try and download them in parallel instead.
I found the gcloud-aio-storage package, however the documentation is a bit sparse, especially for the download function.
I would prefer to download / store the files in memory instead of downloading to local machine then upload to script.
This is what I've pieced together, though I can't seem to get this to work. I keep getting a timeout error. What am I doing wrong?
Note: Using python 3.7, and latest of all other packages.
test_download.py
from gcloud.aio.storage import Storage
import aiohttp
import asyncio
async def gcs_download(session, bucket_name, file, storage):
async with session:
bucket = storage.get_bucket(bucket_name)
blob = await bucket.get_blob(file)
return await blob.download()
async def get_gcsfiles_async(bucket_name, gcs_files):
async with aiohttp.ClientSession() as session:
storage = Storage(session=session)
coros = (gcs_download(session, bucket_name, file, storage) for file in gcs_files)
return await asyncio.gather(*coros)
Then the way I'm calling / passing in values are as follows:
import test_download as test
import asyncio
bucket_name = 'my_bucket_name'
project_name = 'my_project_name' ### Where do I reference this???
gcs_files = ['bucket_folder/some-file-2020-10-06.txt',
'bucket_folder/some-file-2020-10-07.txt',
'bucket_folder/some-file-2020-10-08.txt']
result = asyncio.run(test.get_gcsfiles_async(bucket_name, gcs_files))
Any help would be appreciated!
Here is related question, although there are two things to note: Google Storage python api download in parallel
When I run the code from the approved answer it ends up getting stuck and never downloads
It's from before the gcloud-aio-storage package was released and might not be leveraging the "best" current methods.
It looks like the documentation for that library is lacking, but I could get something running, and it is working on my tests. Something I found out by looking at the code is that you don’t need to use blob.download(), since it calls storage.download() anyways. I based the script below on the usage section, which deals with uploads, but can be rewritten for downloading. storage.download() does not write to a file, since that is done by storage.download_to_filename(). You can check the available download methods here.
async_download.py
import asyncio
from gcloud.aio.auth import Token
from gcloud.aio.storage import Storage
# Used a token from a service account for authentication
sa_token = Token(service_file="../resources/gcs-test-service-account.json", scopes=["https://www.googleapis.com/auth/devstorage.full_control"])
async def async_download(bucket, obj_names):
async with Storage(token=sa_token) as client:
tasks = (client.download(bucket, file) for file in obj_names) # Used the built in download method, with required args
res = await asyncio.gather(*tasks)
await sa_token.close()
return res
main.py
import async_download as dl_test
import asyncio
bucket_name = "my-bucket-name"
obj_names = [
"text1.txt",
"text2.txt",
"text3.txt"
]
res = asyncio.run(dl_test.async_download(bucket_name, obj_names))
print(res)
If you want to use a Service Account Token instead, you can follow this guide and use the relevant auth scopes. Since Service Accounts are project-wise, specifying a project is not needed, but I did not see any project name references for a Session either. While the GCP Python library for GCS does not yet support parallel downloads, there is a feature request open for this. There is no ETA for a release of this yet.

Input Messages in Discord.py

I'm trying to find a way to program my bot to clear a specific amount of messages in a channel. However, I do not know how to get my bot to run it's code based on the user's input data. For example, let's say I'm a user who wants to clear a specific amount of messages, like let's say 15 messages. I want my bot to then clear 15 messages exactly, no more, no less. How do I do that?
if message.content == "{clear":
await message.channel.send("Okay")
await message.channel.send("How many messages should I clear my dear sir?")
This is legit all I got lmao. I'm sorry that I'm such a disappointment to this community ;(
Using a on_message event, you'd have to use the startswith mehtod and create a amount variable which takes your message content without {clear as a value:
if message.content.startswith("{clear"):
amount = int(message.content[7:])
await message.channel.purge(limit=amount+1)
However, I don't recommend using on_message events to create commands. You could use the commands framework of discord.py. It will be much easier for you to create commands.
A quick example:
from discord.ext import commands
bot = commands.Bot(command_prefix='{')
#bot.event
async def on_ready():
print("Bot's ready to go")
#bot.command(name="clear")
async def clear(ctx, amount: int):
await ctx.channel.purge(limit=amount+1)
bot.run("Your token")
ctx will allow you to access the message author, channel, guild, ... and will allow you to call methods such as send, ...

Discord Bot cannot send messages because send is not an attribute of context

I am re implementing my commands the correct way accordingly to the documentation, using context and command decorators instead of on_message listeners, transferring my commands over is kind of a pain but the documentation has been rather helpful thankful. Unfortunately I've run into an issue which prevents me from sending messages...
Before the move, the way I would send messages was like this
#client.event
async def on_message(message):
if message.author.id in AdminID:
await client.send_message(message.channel. 'message')
Unfortunately this does not work on the new format because there is no message argument to get information from, what you have to do is use is the ctx (context) argument instead which looks something like this according to the documentation
#bot.command()
async def test(ctx, arg):
await ctx.send(arg)
Although the bot recognizes the command and goes there, i cannot send a message because send is not an attribute of ctx, this code is taken strait out of the documentation, am I missing something? Can someone help me figure this out? Thank you
You're looking at the documentation for a different version of the library than the one you're using.
You're using version 0.16, also called the "async" branch. The documentation for that branch is here
You're reading the documentation for the 1.0 version, also called the rewrite branch.
Your command would look something like
#bot.command(pass_context=True)
async def test(ctx):
if ctx.message.author.id in AdminID:
await client.send_message(ctx.message.channel, 'message')

Discord Python: How to add cooldown command under an event

I have looked at other posts in regards to how to make a cooldown command. One answer did exactly what I wanted, but that was only because I did exactly what they did. Now I want to officially implement the command into my Discord bot. I noticed that the cooldown command that I tested only worked under #client.command rather than #client.event (client is the object). I have all of my commands listed under the event thing, so I need help on how to add the cooldown command without having to rewrite a lot of things. This is what I have so far regarding the cooldown command.
from discord.ext.commands import Bot
from discord.ext import commands
client = Bot(command_prefix="?")
#client.event
#command.cooldown(5, 30, commands.BucketType.user)
async def on_message(message):
if message.content.upper().startswith("?HELLO"):
msg = 'Hello {0.author.mention} :banana:'.format(message)
await client.send_message(message.channel, msg)
#on_message.error
async def on_message_error(self, error, ctx):
if isinstance(error, commands.CommandOnCooldown):
msg = ':exclamation: This command is on cooldown, please try again in {:.2f}s :exclamation:'.format(error.retry_after)
await self.send_message(ctx.message.channel, msg)
I'm just using one command as an example to show what kind of format I have. I get the error with the #on_message.error (it was a trial an error thing, so I didn't expect it to work). I want to set a cooldown for 30 seconds after 5 consecutive attempts of the same command as well as an error message for the bot to say in response with the timer. I don't really want to have to rewrite the whole thing just for one command to work considering how far I've come to make this bot :/
You should add:
#commands.cooldown(1, 30, commands.BucketType.user)
This will add a ratelimit of 1 use per 30 seconds per user.
You can change the BucketType to default, channel or server to create a global, channel or server ratelimit instead, but you can only have 1 cooldown on a command.
This will also cause a CommandOnCooldown exception in on_command_error

Resources