I need to give users the option to download a report into Excel.
The report includes about 70,000 records (and about 50 fields).
The report generates within about 1.5 minutes, but when the user tries to download, nothing happens.
(In Explorer, a second tab opens and the progress wheel keeps turning...)
I have tried on a filtered list of about 10,000 and it downloads quickly without issue.
However, at 30,000 it already has issues.
I tried from this from the server itself, so network issues were ruled out.)
I also tried sending out the report by email subscription, but this also failed.
Is there a way to go around this size limitation?
Or to give the user similar flexibility on the reports server itself, as he would have in Excel, without building in every possible filter into the report itself. (Probably too big a question here.)
Related
I need to access about 9 files in a particular sharepoint sub-folder for my powerbi visualisation. Each file holds different data and I need them as separate tables.
I tried the below approach as I felt connecting to the share point folder is really slow.
Sharepoint_Folder_Query > This connects to share point site. And filters for the subfolder. This uses "Sharepoint.files" as I was not able to use sharepoint.contents in subfolder since the files are in a subfolder.
File_Query > This references the above "Sharepoint_Folder_Query" and picks up the file it needs. There is 9 File_Query(s), one for each file
Data_Query > There are again 9 data queries each referencing the respective "File_Query" and performs additional manipulation on the data. These are the tables used in my visualisations.
My idea on this was that since connecting to sharepoint takes a lot of time, I wanted to connect just once and use the reference from then on.
But right now my refresh is taking almost 1 hour... During the refresh I could see each of my queries are trying to connect to share point... I could see the message "Waiting for https://..." under all my queries. Not sure what I did wrong.
For comparison, initially I had the files in one-drive and just used the Data_Query section and created all the visualisations. It then took me only 30 seconds to refresh.
So my question. Is something wrong with my approach? if yes what? Also is there a better way to do this and reduce the refresh time.
I currently have an excel based data extraction method using power query and vba (for docs with passwords). Ideally this would be programmed to run once or twice a day.
My current solution involves setting up a spare laptop on the network that will run the extraction twice a day on its own. This works but I am keen to understand the other options. The task itself seems to be quite a struggle for our standard hardware. It is 6 network locations across 2 servers with around 30,000 rows and increasing.
Any suggestions would be greatly appreciated
Thanks
if you are going to work with increasing data, and you are going to dedicate a exclusive laptot for the process, i will think about install a database in the laptot (MySQL per example), you can use Access too... but Access file corruptions are a risk.
Download to this db all data you need for your report, based on incremental downloads (only new, modified and deleted info).
then run the Excel report extracting from this database in the same computer.
this should increase your solution performance.
probably your bigger problem can be that you query ALL data on each report generation.
In one of my sharepoint sites I have a document library with 7,000 document sets, if I count the files inside there are 21000 files in total.
In the beginning we had some views, but when they growth we had list view threshold issues. What I did was to remove some of those views and use search results webparts to get the results the user wants. For me incrementing the threshold is not a solution because this document library grows fast, (2K per month)
This solved the problem for some time.
However, some users do require to export to excel to do pivots based on this data, the only way I can think of is using reporting services in integrated mode with sharepoint, because I can export reports to excel and then they can do pivots.
The question is, will I have the same threshold problem when I make a report based on list data?
What other options do I have?
I have exported files to excel with 600,000+ rows. If the data you are pulling out reaches a certain size you will have to transition to .csv files as excel too has a threshold. The main issues you will run into on very large datasets are timeout issues that can be managed by configuring your http and ssrs timeouts, however, this will lead to other issues including long running reports of 15+ minutes and bandwidth usage.
I would recommend testing your scenario with a simple report to pull your data and see where the limit is reached. Also, look into some filtering mechanisms using parameters to reduce the amount of data returned to the client. If it becomes unmanageable then you may want to look into SSIS or some of the data-warehousing features. SSRS also has cached reporting that can place the processing burden to off hours if real-time data is not a necessity.
I have an attendance recording system that has 2 databases, one for current, another for archiving. The server processes attendance records, and puts records marked completed into the archive. There is no processing done in the archive database.
Here's the issue. One of the requirement was to build a blank record for each staff every day, for which attendance records are put into. The agent that does this calls a few procedures and does some checking within the database. As of current, there are roughly 1,800 blank records created daily. On the development PC, processing each records takes roughly 2 to 3 seconds, which translates to an average of an hour and a half. However, when we deployed it on the server, processing each records takes roughly 7 seconds, roughly translates into 3 and a half hours to complete. We have had instances when the agent takes 4.5 to 5 hours to complete.
Note that in both instances, agents are scheduled. There are no other lotus apps in the server, and the server is free and idle most of the time (no other application except Windows Server and Lotus Notes). Is there anything that could cause the additional processing time compared on the development PC and the server?
Your process is generating 1800 new documents every day, and you have said that you are also archiving documents regularly, so I presume that means that you are deleting them after you archive them. Performance problems can build up over time in applications like this. You probably have a large number of deletion stubs in the database, and the NSF file is probably highly fragmented (internally and/or externally).
You should use the free NotesPeek utility to examine the database and see how many deletion stubs it contains. Then you should check the purge interval setting and consider lowering it to the smallest value that you are comfortable with. (I.e., big enough so you know that all servers and users will replicate within that time, but small enough to avoid allowing a large buildup of deletion stubs.) If you change the purge interval, you can wait 24 hours for the stubs to be purged, or you can manually run updall against the database on the server console to force it.
Then you should run compact -c on the NSF file, and also run a defrag on the server disk volume where the NSF lives.
If these steps do improve your performance, then you may want to take steps in your code to prevent recurrence of the problem by using coding techniques that minimize deletion stubs, database growth and fragmentation.
I.e., go into your code for archiving, and change it so it doesn't delete them after archiving. Instead, have your code mark them with a field such as FreeDocList := "1". Then add a hidden view called (FreeDocList) with a selction formula of FreeDocList = "1". Also go into ever other view in the database and add & (!(FreeDocList = "1")) to the selection formulas. Then change the code adds the new blank documents, so that instead of creating new docs it just goes to the FreeDocList view, finds the first document, sets FreeDocList = "0", and clears all the previous field values. Of course, if there aren't enough documents the FreeDocList view, your code would revert to the old behavior and create a new document.
With the above changes, you will be re-using your existing documents whenever possible instead of deleting and creating new ones. I've run benchmarks on code like this and found that it can help; but I can't guarantee it in all cases. Much would depend on what else is going on in the application.
We have a bunch of reports on SharePoint, using SQL Reporting Services.
The statistics reports - the ones that aggregate data and display few hundreds to a few thousands records are loading fine.
However, we also have reports that display raw records from database. These reports usually have tens or hundreds of thousands of records. Sometimes even millions. And most of the times, they do not load but throw OutOfMemory errors.
The queries for those reports are very simple selects with some where conditions (sometimes, another few small tables might be joined on the huge one). In SQL Server Management Studio the query completes in 5-10 seconds.
I'm frustrated, because the clients are asking for the report, but I can't find any solutions to this (I googled a lot, but the best advice I could find was "get rid of the report or try to minimize the amount of data in it" which doesn't really solve anything - clients insist that they need the ENTIRE report.)
Is it possible to solve this somehow?
Move to 64 bit for the reporting server?
Chances are the users need the ENTIRE report because they are scraping the data off into excel or some other format and using it elsewhere. If you can get away with it, coding a webpage or similar that displays the query in a simple text format/csv may be more effective than a report.
I.e. The best advice you can find is the best advice.