Im struggling with Asset\Image. Is there a way to prevent given image from saving on disc and read from given location instead?
Currently Im creating Image like so
$asset = new Asset\Image();
$asset->setFilename($location);
$asset->setData(file_get_contents($location));
$asset->setParent(Asset::getByPath("/"));
$asset->save();
And obviously it gets saved on in the public/var folder.
Unfortunatelly I cannot afford to do so, because there is like hundreds of GBs of photos.
Is there a way to use location to save and later read image content from saved location?
I think this is not possible by php. You can achive your goal by saving all assets in another location and than replace the public/var folder with a symlink
mkdir /your/asset/storage
cp public/var/assets /your/asset/storage
Danger zone starts here, be sure to have a good backup of all assets
rm -r public/var/assets
ln -s /your/asset/storage public/var/assets
This is only an example and might differ on your system
What was your idea to save disk space? The file takes space anyway, on the disk directly or on the disk managed by Pimcore.
If you have hundreds Gb then check please if those images are Assets or their versions (created on each save).
If they are versions then you have options:
limit versions amount/time (see System Settings -> Assets) or in pimcore/system.yml
pimcore:
assets:
versions:
days: null
steps: 3
disable versioning before to save
Version::disable();
$asset->save();
Version::enable();
And remember that one day you can grow big and need more then 1 fileserver. Pimcore Assets can already manage that, but how will you synchronise assets files?
Related
I recently reinstalled Ubuntu 22.04.1 from scratch.
Since my computer has a 120GB SSD and a 1TB HDD I decided to mount / on the SSD (together with a 20GB swap area) but to keep my files in the HDD.
To do so I set the HDD to mount at startup on /mnt/data/ and I created there all the home directories (Desktop, Pictures, Documents, ect...). Then I removed the default home directories and created symlinks pointing to the HDD this way:
ln -s /mnt/data/Downloads $HOME/Downloads
I did the same for Documents, Pictures, Music, Videos, Public, Models and Desktop.
At first it worked fine but after a little while all of the links where broken and the output of file was:
jacopo#desktop-jacopo:/mnt/data$ file Videos
Videos: broken symbolic link to /home/jacopo/../../../../../mnt/data/Videos
This happened for every link that I just created.
I repeated the process a couple of times with the same outcome but I found out that it doesn't happen if the source directory is not empty. (e.g. if I put some file inside /mnt/data/Downloads the link to /mnt/data/Downloads never breaks).
It seems like somehow the source directories automatically become links that point to the actual link creating a circular link.
So my first question is what's happening? What did I do wrong to create such unstable links?
But I also wonder if my approach to save space on the SSD is right. (I didn't want a separate home partition on the HDD because I thought it would slow down the start of every program)
To be a little clearer this is my final setup (that breaks):
/
/mnt/
/mnt/data/
/mnt/data/Desktop
/mnt/data/Documents
/mnt/data/Downloads
/mnt/data/Models
/mnt/data/Music
/mnt/data/Pictures
/mnt/data/Public
/mnt/data/Videos
/home/
/home/jacopo/
/home/jacopo/Desktop -> /mnt/data/Desktop
/home/jacopo/Documents -> /mnt/data/Documents
/home/jacopo/Downloads -> /mnt/data/Downloads
/home/jacopo/Models -> /mnt/data/Models
/home/jacopo/Music -> /mnt/data/Music
/home/jacopo/Pictures -> /mnt/data/Pictures
/home/jacopo/Public -> /mnt/data/Public
/home/jacopo/Videos -> /mnt/data/Videos
When I run the command in the terminal back to back, it doesn't sync the second time. Which is great! It shouldn't. But, if I run my build process and run aws s3 sync programmatically, back to back, it syncs all the files both times, as if my build process is changing something differently the second time.
Can't figure out what might be happening. Any ideas?
My build process is basically pug source/ --out static-site/ and stylus -c styles/ --out static-site/styles/
According to this - http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
S3 sync compares the size of the file and the last modified timestamp to see if a file needs to be synced.
In your case, I'd suspect the build system is resulting in a newer timestamp even though the file size hasn't changed?
AWS CLI sync:
A local file will require uploading if the size of the local file is
different than the size of the s3 object, the last modified time of
the local file is newer than the last modified time of the s3 object,
or the local file does not exist under the specified bucket and
prefix.
--size-only (boolean) Makes the size of each key the only criteria used to decide whether to sync from source to destination.
You want the --size-only option which looks only at the file size not the last modified date. This is perfect for an asset build system that will change the last modified date frequently but not the actual contents of the files (I'm running into this with webpack builds where things like fonts kept syncing even though the file contents were identical). If you don't use a build method that incorporates the hash of the contents into the filename it might be possible to run into problems (if build emits same sized file but with different contents) so watch out for that.
I did manually test adding a new file that wasn't on the remote bucket and it is indeed added to the remote bucket with --size-only.
This article is a bit dated but i'll contribute nonetheless for folks arriving here via google.
I agree with checked answer. To add additional context, AWS S3 functionality is different than standard linux s3 in a number of ways. In Linux, an md5hash can be computed to determine if a file has changed. S3 does not do this, so it can only determine based on size and/or timestamp. What's worse, AWS does not preserve timestamp when transferring either way, so timestamp is ignored when syncing to local and only used when syncing to s3.
I have a video files in:
/home/private/movies/video1.mkv
/home/private/movies/video2.mkv
/home/private/movies/video3.mkv
I have hardlinks to those mkv files in:
/home/minidlna/videos/video1.mkv
/home/minidlna/videos/video2.mkv
/home/minidlna/videos/video3.mkv
My minidlna share is:
/home/minidlna
The video files show up on the minidlna cilent (my TV) after I do a full rescan of the minidlna share, however, they don't show up if I create new hardlinks with the inotify interval set really low.
The files do show up if they are not hardlinks.
My guess is that there seems to be a problem with minidlna and the way it processes the 'filesystem changes' using 'inotify'. Perhaps a hardlink isn't necessary a 'change' to notify minidlna.
My video library is rather large and continually doing rescans seems very inefficient and takes a long time. I would appreciate if someone can shed some light on this or have a workaround.
I'm running minidlna version 1.1.4
It appears it is indeed a problem with minidlna.
Depending on your use case, maybe you can create the new video file in the minidlna directory and make the one in your private movies a hardlink. The resulting filesystem will be the same, but now the first operation minidlna sees should be a full-fledged create, and therefore work.
Looks like there's no workaround to my exact problem and unfortunately my setup doesn't allow reversing the minidlna share <> hardlink directory.
The only solution I found was to rebuild minidlna RPM with IN_CREATE in inotify.c (more details here - http://sourceforge.net/p/minidlna/bugs/227/)
Hopefully Readynas makes that the default for future builds.
I have a problem in Windows Azure. I'm storing temporary files in local storage. After a certain time i get a System.IO.IOException: There is not enough space on the disk.
So I have read some articles about it and microsoft themself recommends to catch the error and try to clear the files. So my question at this point is how is the best way to accomplish this?
At the moment I would try this but I don't know if this is the best approach:
public static void ClearTempFolder(string localStorageName)
{
System.IO.DirectoryInfo downloadedMessageInfo = new DirectoryInfo(RoleEnvironment.GetLocalResource(localStorageName).RootPath);
foreach (FileInfo file in downloadedMessageInfo.GetFiles())
file.Delete();
foreach (DirectoryInfo dir in downloadedMessageInfo.GetDirectories())
dir.Delete(true);
}
Thanks for your help.
If you're happy for all the files to go - then, yes, that should work fine. You may want to trap for exceptions that will be thrown if a file is still open.
However, it may be better to examine your code to see whether you can remove the temporary file immediately when you've finished with it.
Check out http://msdn.microsoft.com/en-us/library/windowsazure/hh134851.aspx
The default TEMP/TMP directory limit is... 100MB! Even if you have 200GB+ local storage.
Your solution should be two fold:
1) Clean up temporary files when you're done with them (if you write a file to the temp folder, delete it when you're finished with it)
2) Increase the local storage size (as above) so you can store files larger than 100MB on temporary disk storage
THE SCENARIO
I'm developing a Root FS for some embedded Linux device. It is sitting on the host, exported via NFS and my development board mounts is under "/".
The workflows that I need are:
- to share my FS to other developers(they have with their own dev. boards)
- to backup my Root FS onto some "server"
- to deploy my Root FS onto flash-disks or other media
- track changes in specific files in my Root FS, branching&merging,roll back etc.
Guys, this seems to me as a Version Control scenario, and I even use git.
THE PROBLEM
As you know Git(and svn/mercurial/bazaar too !) 1) does not store special files (device files under /dev etc.) 2) does not store file owners and permissions.
I want to store everything and AS IS.
THE QUESTION:
Do you know some VCS that will do the job ?
Or may be you know about another (but simple) solution for doing my scenarios ?
IS IT A COMMON PROBLEM...
I believe that it is, because till now I've heard about scripts/hooks/custom soft that everybody(!) works out for his purposes. All I need is an all-eating-VSS
Thank you !!
Having done something similar (developing firmware for an embedded Linux OS), I've found that it's better to put device file creation into a script called by your build system, rather than to store device files directly on the development machine. Then the script that creates the files goes into version control, instead of the files themselves, and (BONUS) you don't need to be root to modify them. Run the build through fakeroot instead.
I suppose this doesn't directly answer your question, but it may be worth thinking about revising your development model. It's NEVER a good idea to run your build as root, because what happens if you accidentally have a "/" in front of a common path? You may inadvertently replace /bin with a whole bunch of links to busybox built for a different architecture.
This is the tool for you:
http://fsvs.tigris.org/
it has svn backend.
I know this seems a little obvious, but as you haven't mentioned it: Have you considered mechanisms to put all your special files into a regular file, like, for example, into a tar archive? You could store that just fine with any version control system, and as filesystems have lots of binary data anyway diffs between two revisions of a full root filesystem aren't that useful anyway, so you might even not lose too many of the features your version control system provides.
initramfs is a good answer to the userid groupid, permissioon problem. In your kernel source directory, there is scripts/gen_initramfs_list.sh.
This script allows you to build an initramfs archive from several sources. You can for example, specify :
a directory : The files and directory found in this base directory will be at the root of your file system.
a file liste : it is a text file, very useful to create directory, files and special device files. See example below
If you develop as non root, and your rootfs is in rootfsdir, then probably the file in rootfsdir are owned by you. gen_initramfs_list can translate your uid, gid into 0, 0. Here is an exemple command line :
gen_initramfs_list -u $MYUID -o initramfs.gz rootfsdir/ device.txt
Where device.txt contains :
# This is a very simple, default initramfs
dir /dev 0755 0 0
nod /dev/console 0600 0 0 c 5 1
dir /root 0700 0 0
# file /kinit usr/kinit/kinit 0755 0 0
# slink /init kinit 0755 0 0
Then you can use standard version control for your rootfsdir content, and add the device.txt file under version control, and here you are : content and file attribute are versionned :).
I don't know if you can change the permission and uid/gid of a file in a directory source via a filelist source, but this would be a logical feature.
Of course you can start with minimal root fs, from which you mount your existing nfs_export.
It is a common problem, and gen_initramfs_list is the tool to solve it.
Why not just use rsync? Something like rsnapshot (http://www.rsnapshot.org) will do what you want. Alternatively Unison (http://www.cis.upenn.edu/~bcpierce/unison/) is explicitly designed for this - it describes itself as a cross-platform file synchronisation technology with version control, and might be what you need.