stackexchange redis, reading large file = timeouts - azure

Question

I have an application that uses a large in memory file (just under 2gb).
I'm trying to use redis lists (in Azure) as the storage (vs SQL). Building the list in redis is pretty fast, I can load the redis list in about 5 minutes but then I need to read from the list into the application.
This is extremely slow, I've tried increasing the threads, extending synctimeout etc to no avail.
ThreadPool.SetMinThreads(200, 200);
I'm using a C# implementation of redis list that I discovered online, I'm passing this to the code that builds the in memory collection via foreach loop. Internally this is how it processes the data (I've omitted the rest of the class)
public class RedisList<T> : IList<T>
{
private static ConnectionMultiplexer _cnn;
private readonly string _key;
private static readonly Lazy<ConnectionMultiplexer> LazyConnection = new Lazy<ConnectionMultiplexer>(() => ConnectionMultiplexer.Connect(ConfigurationManager.AppSettings["AzureRedisCacheUrl"]));
public ConnectionMultiplexer Connection => LazyConnection.Value;
public RedisList(string key)
{
this._key = key;
_cnn = Connection;
}
public IEnumerator<T> GetEnumerator()
{
for (var i = 0; i < this.Count; i++)
{
yield return Deserialize<T>(GetRedisDb().ListGetByIndex(_key, i).ToString());
}
}
}
Is there a more efficient way of reading in the data? Am I insane to do it this way? :D Thanks

Answer 1

Am I insane to do it this way?
Absolutely. Redis isn't designed to store large binary data like < 2GB files (or even 100MB files).
Redis is about indexing small chunks of data to retrieve them later in a very optimized and efficient way in terms of both CPU and memory. Remember that Redis is an in-memory database, and the fact that snapshots its data to disk (for example in RDB files) doesn't mean that the data source is your RAM memory.
Instead of storing this large binary data on Redis, just use Redis as file index, and leverage its data structures to get back to them in a breeze.
OP said in some comment:
Hi Matias, the redis list is a collection of individual files ranging
from a few kb to around 5mb. The legacy application I'm working on
loads all this into a huge static object on application start (I know,
awful). The original version was an in memory file loaded from SQL,
this proved too slow when moving to azure so we need a faster
intermediary store of the data
Anyway, Redis isn't designed to be an in-memory file storage.
I would say that you should take a look at memory-mapped files, where you can even load all files into a memory mapped file, and get them using indexes (for example, from byte 0 to 243843, is file1, and so on). This should improve overall performance and you won't need to use wrong tools for the job.