Closed. This question is not about programming or software development. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 3 months ago.
Improve this question
I have such a task
I have file storage, about 50GB
Almost all files are 100KB -> 3MB.
And there are tracing files of ~25mb - but these files are downloaded very often
My task is to configure the software in such a way as to ensure maximum download performance from my Linux Server
Server:
System: Ubuntu 20
Disk: SSD NVMe
RAM 64 GB
CPU: 12
Internet: 1 Gbit/s
I tried the following combinations
Nginx
Varnish + Nginx
but I am facing the problem that the file download speed is up to 1 MB per second - even though I changed the settings in Nginx
Average download time 16 sec = 25MB - from my server
2-3 seconds - 25MB - from the Firebase server
What software can be suitable for solving my problem? or in what direction do I need to look?
Varnish Cache, the open source version of Varnish, can handle up to 100 Gbps of throughput on a single machine. Since your NIC is limited to 1 Gbps.
If you want the 50 GB catalog to be stored in the cache at all times, ensure you assign at least 50 GB to the varnishd runtime process by setting the -s option to the right value.
Please keep in mind that Varnish also needs memory to process the workloads on the various threads.
Make sure that the VCL configuration in /etc/varnish/default.vcl is properly configured to serve these files from the cache.
If that files are static, you can assign a very long TTL to each cached object to ensure they are always served from the cache.
If it turns out that the catalog of files grows and exceeds the 50 GB limit you set, either increase the size of the cache, or accept the fact that the least recently requested objects are going to be removed from the cache to free up space for new objects.
Sizing your cache is a tradeoff and also depends on the frequency with which certain objects are fetched. The impact of a cache miss on your origin web server is also a factor to keep in mind.
In general I would advise you to assign about 80% of your server's memory to Varnish for caching. Unless you have crazy traffic spikes that go beyond the basic threading settings of Varnish, you'll be fine.
This is a very basic and high-level answer, but it validates Varnish as a candidate to accelerate the 50GB of content you've referred to in your question.
For me the best solution turned out to be the solution https://github.com/minio/minio
After installation and basic configuration, the installation on my server turned out to be quite productive
Now I have ordered additional options for myself from the provider
Such as 10G internet channel as well as additional ssd
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I read a lot about the importance of having at least 2 websites instances in Azure, one reason being that MS will only honour it SLA, if there is, due to being able to patch one server while having the other available.
However we current have strict budgets, and currently have 1 medium server with the bigger RAM. I have always believed that bigger server with more RAM is always better. Also 2 cores on the same machine may be quicker as well.
We have noticed the odd recycle, but it is too early to say whether this is due to MS patching.
Assume my application is a MVC3/EF5/SQL Azure app with 10 user concurrency, and processing is straigtforward, ie simple DB queries etc.
In the context of Windows Azure, assuming a budget limit, would 1 medium(2 x 1.6Ghz cores and 3.5 GB RAM) server be better than 2 small(1 x 1.6GHz Core and 1.75GB RAM) web server instances.
Thanks.
EDIT 1
I noticed this question has attracted 2 votes for being opinion based. The question is designed to attract reports from real experience in this area, which of course informs opinion. This is hugely valuable for my work, as also others.
EDIT 2
Interesting about SLA. I was concerned about when MS does an update, then one instance would disappear while this occurred. So what would happen in this case? Does Azure just clone up another instance? Also what happens in situations where one instance is working on a slower process, it might be waiting for something like a DB transaction. With 2 instance the LB would redirect to instance 2. Logically this sounds superior. It will still work with session vars as MS has implemented "sticky sessions".
I am intrigued that you recommend going with a "small" instance. 1.75GB RAM seems so tiny for a server, and 1 core at 1.6GHz. Need to do some memory monitoring here. Out of interest, how many times would the main application dlls load into RAM, is it just the once regardless of numbers of users? May be a basic question, but just wanted to check. Makes you think when one's laptop is 16GB and 8 cores (i7). However I quess there is a lot of different bloating processes going on a laptop, rather than many fewer and small processes on the server.
Unless your app is particularly memory hungry, I would go for a single small and configure the autoscale to start more servers as needed. Then just keep an eye on the stats. You can have a look at how much memory you are currently using; if it's less than what you get with a small instance you don't get any benefit from the extra RAM.
The SLA for Websites does not require two instances, that rule applies only to Cloud Services.
I have found that you can do a surprisingly large amount of work on single, small instances; I have several systems in that kind of setup which only use a few pct of capacity, even at hundreds of requests per minute. With 10 users you are unlikely to even have IIS use more than one thread, unless you have some very slow responses (I'm assuming you are not using async) so the second core will be idle.
For another example, look at Troy Hunts detailed blog about haveibeenpwned.com which runs on small instances.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
I am new to Azure. I have created a VM. and stored some very important files on temp storage, but after few days the temp disk is formatted. is there any option I can get my data back ??
Eshant
Just wanted to provide you with clear guidance regarding the disks and why you lost data. There are 3 Disks with in Azure VM, the are as follows
C Drive 127 GB Dedicated for OS will persist after reboot. This disk is dedicated for OS and shouldn't be used for any other purpose.
D Drive is Temporary Drive is only intended for storing temporary data. As you notice it only has page file on it and is not recommended for storing data because it is wiped clean on Stop and start, resize of VM , un/planned maintenance, and service healing. One key benefit of this disk is performance. I/O performance for temporary disks is higher than the IO permanence to OS disks, Data Disks. The size of the disk varies with VM Size. In your case the data is lost and cannot be recovered.
Data Disk. You need to add Data disks for any type of custom storage that needs to be persisted. Another point being ,the difference between OS and data disks is that, while both reside in blob storage, the host caching settings are different by default - OS disk is Read/Write host caching by default, data disks are None host caching by default.
One key point is , the C:\ and D:\ cost is included in the VM Price , Data disk will be charged on actual usage. Say if you allocate 100 GB and use only 10 GB. Then you be charged only for the 10 GB.
Regards
Krishna
No, there's no way to recover your data.
From this Microsoft article: Understanding the temporary drive on Windows Azure Virtual Machines
http://blogs.msdn.com/b/wats/archive/2013/12/07/understanding-the-temporary-drive-on-windows-azure-virtual-machines.aspx
Is there a way to recover data from the temporary drive?
There is no way to recover any data from the temporary drive.
How to Calculate the maximum capacity of a Linux Apache web server, ie) To serve maximum request parallel.
Or
To perform "N" request parallely, what is the configuration required?
I need to optimize my Web server so that it needs to response 1500 users parallel.
The question is not answered so easily. It depends on the size of your RAM, your connection, CPU, the amount of memory each request consumes, the amount of memory the server side script (php/ruby whatever) consumes and the mode apache (mpm_worker, prefork, itk etc.) and the script execution (cgi, mod_php, fcgi, suphp etc.) run in. It also depends on the configuration of each of those modules, at least partly. Last but not least, a caching system like varnish can take of 90% or more of the load if used properly.
Basically try running a test from another (or 2, 3, 4) web server(s) and see if your system will serve the pages in an acceptable time.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
Embedded system, no swap, kernel v2.6.36, memory compaction enabled.
Under heavy usage, all the RAM is tied up in cache. Cache was using about 70M of memory. When a user space process allocates memory, no problem, cache gives it up.
But there's a 3rd party device driver that seems to try to allocate a physical 5th order page, and fails with OOM. A quick look at buddyinfo confirms this... no 5th order page available. But as soon as I drop cache, plenty becomes available and the device driver no longer OOM.
So it seems to me that virtual memory allocation will trigger cache drop, but physical memory allocation will not? This doesn't make sense, because then kernel modules are likely to OOM when memory is tied up in cache, and this behavior seems to be more detrimental than slow disk access from no caching.
Is there a tuning parameter to address this?
Thanks!
So here's what's going on. I still don't know why high cache use is causing kernel modules to OOM. The problem is in 3rd party code that we don't have access to, so who knows what they're doing.
I guess one can argue if this is by design, where non-critical disk cache could take all available free memory, and cause kernel modules to OOM, then IMHO, maybe disk cache should leave something for the kernel.
I've decided to instead, limit the cache, so there is always some "truly free" memory left for kernel use, and not depend on "sort of free" memory tied up in cache.
There is a kernel patch I found that will add /proc/sys/vm/pagecache_ratio so you can set how much memory the disk cache could take. But that was never incorporated into the kernel for whatever reason (I thought that was a good idea, especially if disk cache could cause kernel OOM). But I didn't want to mess with kernel patches for maintainability and future-proofing reasons. If someone is just doing a one-shot deal, and doesn't mind patches, here's the link:
http://lwn.net/Articles/218890/
My solution is that I've recompiled the kernel and enabled cgroups, and I'm using that to limit the memory usage for a group of processes that are responsible for lots of disk access (hence running up the cache). After tweaking the configuration, it seems to be working fine. I'll leave my setup running the stress test over the weekend and see if OOM still happens.
Edit
I guess I found my own answer. There are VM tuning parameters in /proc/sys/vm/. Tune-able settings relevant to this issue are: min_free_kbytes, lowmem_reserve_ratio, and extfrag_threshold.
This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.
Closed 10 years ago.
We've recently upgraded (migration path) from TFS/Sharepoint to TFS2010/Sharepoint2010.
Most things went well, but there were a few issues that became immediately apparent.
TFS was noticeably slower (as pointed out by the entire dev team). Basically all "get latest", and query operations were more sluggish. Starting VS2010 SP1 is also really slow with loading all the projects (40+) on my machine. A refresh after that is not normally a problem. Even though other people may only have 3-4 projects open at the time, they too noticed the "working..." delay.
Sharepoint was definitely much slower. The "Show Portal" takes forever to load, and the basic editing is slower too.
Work items occasionally "time out" for no reason, and end up in a "connection lost" error. It's normally while creating a new work item, and a redo of the same command works fine. It happens even during bulk work item creation, but the timing is random.
The server runs on Windows 2008, 12 GB, and plenty of CPU power (QuadCore). The IIS connectionTimeout is set to 2 minutes (default), I've played with the MinBytesPerSecond which is set to 240 by default (I've set it to 42 as well, but no joy), and I understand that VS 2010 in general might be a bit slower than its 2008 counterpart, but even then. No processors are maxed out. There are lots of MSSQLSERVER info logs in the Event Viewer though (I just noticed this - not sure if this is a problem). I've also changed the defaultProxy setting in the devenv.exe file - no joy there either.
It's too late for a downgrade. ;)
Has anyone experienced similar problems after the upgrade?
I would love to hear from ya! :o)
We experienced performance issues after upgrading from TFS 2008 to 2010 but it is much better now. We have learned that the Antivirus and SQL Server configurations are critical. In a virtualized environment store performance is key too. We have about 100 TFS users in a 2 tier Server setup.
The SQL server has it's default memory setting set as follows:
1 - SQL Server max memory 2TB
2 - Analysis Services max memory 100%
With those settings, our 8GB SQL machine was unusable.
Now we have:
1 - SQL Server max memory 4GB
2 - Analysis Server Max memory 15%
Now the performance is ok but not great.
The Antivirus Exclusions have to configured too. Basically excluded all the data and temp directories.
As our TFS setup is virtualized we are in the process of adding more storage hardware to have better disk performance. That should definitely solve our performance issues.
Simon
are all components installed on one machine? Is SQL layer also installed on that machine? Is the machine virtualized?
It's always better to install SQL layer on physical hardware than installing it virtually. SharePoint 2010 requires 4 gigs of RAM. To ensure that SharePoint is usable you should size the WFE with at least 8 gigs of RAM.
Our TFS was also slow with 4 gigs so I've added another 4 gigs. With this setup the entire environment is right now really fast.
Let's summarize
SQL: physical installation w/ 12GB RAM, Quad Core (duplicated for failover)
SharePoint: virtualized w/ 8GB RAM, Quad Core
TFS: virtualized w/ 8GB RAM, Quad Core
Both SharePoint and TFS are generating heavy load on the database. I've a showcase machine running on my Elitebook as HyperV image. The image has about 12 gigs of ram and is running on an external SSD but it is a lot slower than our productive environment.
I hope my thoughts and setups are helpful.
Thorsten