Hey Guys
At the moment I have a NodeJS webapp in the making which scrapes a website for data. Specifically, this webapp scrapes images for the purpose of downloading them. For example, all the image permalinks are scraped from the reddit front page. They are then sent to the client to download individually. My issue is with the website I am scraping there can be thousands of images.
This provides a horrible user experience if 1000+ images are downloaded to the download folder.
As a result I have two options.
A) Download to a temporary folder on the server. Zip. Send to client for download. Delete from server
B) Download files to browser cache. zip. download to specified download directory.
My question to you is this; Is option B even possible?
I am relatively new to this entire process and I can't find anything to actually zip the files in the browser cache. I can implement option A relatively easily however this requires a large amount of bandwidth, something I can find for around $5/MO on DigitalOcean. However this entire project is a learning experience and as a result I would love to be able to manage files in the browser cache instead.
I am using the following NPM Modules:
NodeJS
Express
Cheerio
Request
Further Update
I had come across a plugin for NPM called jsZip: https://stuk.github.io/jszip/
However, I was unaware it could be implemented on the client side as well. This was purely an error on my part. This brings up an interesting issue of WebStorage: https://www.w3schools.com/html/html5_webstorage.asp
the maximum storage size for the session is 5MB
From here I will attempt to implement this answer here: How do you cache an image in Javascript to my current code and will update this answer with the result for anyone else facing this issue.
Related
I’m running a node app on Heroku. It gathers a chunk of data and sticks it into a set of JSON files. I want to POST this data to my Wordpress server (right now it’s on siteground, but moving to WP Engine).
I thought the WordPress REST API would provide what I wanted but after reading the docs I’m not not so sure.
Does anyone have any advice on this? It’s not the kind of thing I’ve done before.
Naturally, I could download the generated files and manually upload them in the right place… but I want it to be automated!
Can anyone point me in the right direction?
Looked at WordPress REST API but don’t think that’s the answer.
An option would be to setup a cron job on your Wordpress site to download the already made JSON files. This would let you download the files via http if they're publicly available, or using an api-key, or SFTP even.
You could also go the opposite way and setup a cron job on the server running your node app and use SFTP to deliver the updated files onto your Wordpress site.
If you are really committed to wanting to use the wordpress rest api, I think the closest one that comes to my mind (not an expert in wordpress) would be that you'd upload the json files as media objects: https://developer.wordpress.org/rest-api/reference/media/.
Ok, I have never seen anything like this before and hoping someone else has. I just finished patching our Dev and Test servers to Nov2017CU (SharePoint 2013). Since then, any solutions that are using JS injection from Site Assets are not updating. I'll make a change to the file, the library reflects that I made the change, but when I attempt to load the page accessing the js file, the changes are not reflected. Hard refreshes and full cache cleans are not affecting it. If I close and reopen my editor (VSCode) my changes are gone. When I look at the version history, the current version doesn't have my changes, but the previous version does. If I try to revert to that version, it doesn't take (still shows the previous version of the file).
Here's where it becomes extra weird. I have deleted the entire file from the library. Reset IIS (heck, I even rebooted the server at one time). It somehow still loads the file. The file is no longer in the library, but the server is still serving it up to the browser. I have confirmed it is not getting it from another location as the Dev tools are showing the file is located in the Asset Library the file was deleted from. Even users who have never accessed the site before are still getting that file in their browser.
This isn't limited to a single site either. I have other developers in different sub sites (same site collection) that are having the same issues.
Anyone seen this before?
Looks like your web application has BLOB cache enabled which is causing files to served from the cache.
There are 2 ways to fix:
1) The heavy handed way would be to flush the BLOB cache using powershell commands mentioned:
$webApp = Get-SPWebApplication "<WebApplicationURL>"
[Microsoft.SharePoint.Publishing.PublishingCache]::FlushBlobCache($webApp)
This will flush all the files in the BLOB. Usually, the files are cached based on the max-age attribute value. So, that is the reason that your files are being served even if you had deleted it from the source.
2) The surgical knife approach would be to append a query string, like (https://sitecollurl/siteassets/app.js?v=1.1), to the file references (usually in master page, page layouts, webpart references, script links etc. wherever it is referenced). When you append a query string to the file, it will force the browser to download the newer version of the file. Would prefer this approach as it will not unnecessarily clear other files from BLOB.
I have built a desktop app using node webkit and need to cache PDF files that are viewed via the App when online so that they are also available offline. I haven't found a solution yet but during testing I noticed that files that I had previously viewed online were available offline even though I haven't written any code for this yet. Therefore these must already be cached automatically. I did a search to find where the files are being saved exactly but couldn't find anything.
Can anyone explain this or point me in the direction of information on this so that I understand how it works and ensure my App can utilise the default behaviour of the browser caching?
********UPDATE***********
I have found a solution to store the PDFs locally, however this isn't my query. I am looking for an explanation as to HOW the PDFs are available when offline without this code I have written. The files must be automatically be stored somewhere otherwise how would they display?
The default caching behavior of node-webkit is controlled by the page-cache property in package.json :
"webkit": {
"page-cache": true
},
Only typical web resources can be cached this way (scripts, style sheets, etc.). To be able to view PDF files offline, you can store them manually.
There are several ways to do that :
Save a file directly to disk (the simple solution, just store the files in App.dataPath)
Use a database
Use Web Storage
Use the application cache
All of these are documented here : Save persistent data in app
The default location to cache your app files is mentioned in your package.json manifest file.When the app is initialized the settings in your manifest files are loaded by default.Since cached files cannot be accessed programmatically,you can overwrite the default files manually.
To get the application’s data path in user’s directory for windows,you can write it in Jason format in your package :
Windows: %LOCALAPPDATA%/
You can read about other cache menthods in node webkit's documentation :
http://docs.nwjs.io/en/latest/References/App/#appclearcache
I would like to develop an app that targets everything from Gingerbread(version 2.3 API 9) to JellyBean(version 4.3 API 18).
The problem:
I need to transfer large images(40 to 50 at a time) either independently or in a zip file without the user having to click on each file being transferred. As far as I can tell I need to use the HttpClient(org.apache) that was deprecated after JellyBean.
Right now the application takes the images and zips them to a zip file prior to uploading. I can create additional zip files, for example if I have 50MB to transfer I can make each zip file about 10MB and have 5 files to be transferred if I have to. I need to transfer these files to a web server. I cant seem to find anything about transferring files after Jellybean. All the searching I've done uses the deprecated commands and the posts are 2-5 years old. I have installed andftp and transferred a 16MB zip file last night that was created by my app, but I really don't want to use that as it will require additional steps from the user. I will try andftp today and setup an intent to transfer the files to see how that works out. Supposedly andftp works until Lollipop(5.0). If there is an easier way please let me know, hopefully I've missed something about transferring files. Is there another way to do this after JellyBean?
I am tearing my hair out with a file weird file upload issue that I have never run into before. For some reason I’m unable to upload images via the file manager (both in the file manager itself and if I upload with a custom field using the “file” fieldtype). Strangely, if I add files directly to any of the file upload directories, and sync the files, everything works fine.
After selecting the file and hitting “upload file” (see 01_choose_file.jpg) the modal window displays the CP homepage in an iframe (see 02_upload_progress.jpg).
Has anyone else seen this? Does anyone know how I can start troubleshooting this?
Background Info:
I’m running EECMS v2.5.2 - Build Date: 20120606 in MAMP (only 2 out of 15 sites I have set up locally are not working)
I have tried uploading images/files using the latest versions of Chrome, Chrome Canary, Safari, and Firefox (OS X 10.7.5)
This issue is showing up on the two latest sites I’ve started dev’ing locally on, no other site (locally or otherwise)
Things I’ve done:
Checked Apache/PHP error logs; they don’t show anything
Confirmed file upload paths and file upload directory settings are correct – I can sync files that i manually move into the various file upload directories
Permissions are fine; image manipulations and thumbnail creation work fine if I manually add files to the upload directories
Tested various other 2.5.2 installs I dev on locally and they work fine (settings on these two new sites are identical to sites that work)
Only a handful of native add-ons are enabled
“Apply XSS Filtering to uploaded files?” setting Yes or No does not make a difference
Huge thanks for any help!
I can't post images so here are links to the images:
01_choose_file.jpg: http://expressionengine.com/?ACT=51&fid=105&aid=16264_Jiof3p0V1gfEEFrpC55G&board_id=5
02_upload_progress.jpg: http://expressionengine.com/?ACT=51&fid=105&aid=16265_mjGH02xK2fIFZJI6kruP&board_id=5
I have sorted this out. I went back through to make sure I disabled all third party add-ons and I had forgotten to uninstall the "Quickee" extension http://devot-ee.com/add-ons/quickee. For now that seems to be the culprit.
I've submitted the bug to Matt (the developer) and it should be patched up soon.
The ExpressionEngine filemanager sends out a AJAX POST request to the following URL:
http://YOUR_ADMIN_CP_URL?S=0&D=cp&C=content_files_modal&M=upload_file
Have you tried loading that URL yourself? You should get a page like this
But maybe EE is trying to POST to a different URL. You can find it by uploading a large file and while it's uploading using Firebug and in the Network tab at the bottom of the list you will find the URL EE is posting to