I have a requirement to retrieve data from share point (I guess it is 2010, but will check with admin if relevant) and generate an excel report/chart. Say we have a bug tracking system in share point. Currently, I could create a view and see some statistics, but I need to plot a graph to see historically (every week) how the number of bugs changed. For example,
get the number of bugs filed in a specific week
do some grouping based on type/severity
based on classification get number of bugs solved that week etc.
If I can get the numbers based on date range, I may use excel to plot the graph.
After some reading, SharePoint object model come close to what I used to work with (Oracle DB). I understand it may be entirely different from tradition db and querying.
Please help me with
What is the best method to approach this?
Is there a good book/resource.
Thanks a lot,
bsr
The easiest apprach would be to LINK to the sharepoint lists using Access 2007 or 2010 and then export the data to Excel for further processling. Of course, you could also write a program that uses CAML query to access the data. Your requirement sound straightforward, unless you need to automate the reporting process, the simplest approach would be to access the lists via an access database.
You could also create a web service via REST that pulls the data directly into Excel.
SharePoint has it's own query language: CAML query, and in theory that could be used to retrieve the list you seek.
And you should be prepared for "some" trial and error.
Tools I used:
http://www.u2u.be/res/tools/camlquerybuilder.aspx
http://spud.codeplex.com/
what I understand from this question is that you have the need to put the SharePoint data to an excel file and this from within the SharePoint site? So it looks to me that you could just create a simple SharePoint web part that consists of one button "generate excel file". So when the user clicks on the button you would just query your SPList object(SharePoint object model) and you would get all the necessary data from the list (SPListItems).
This is the way that I would take. Mind you that this is offcourse custom SharePoint Development (.NET c#). There are lots of books or blogs that described how to create your own web part in SharePoint.
Related
I have a sharepoint and spying works well if records are record 100-200 in a list. As soon as records increase beyond 500, spying becomes slow and takes more than a minute to spy. I have tried using spying with Path property with MatchIndex set to 1. I have so far tried in html mode and after seeing this , I am not confident on AA mode spying. Has anyone faced such challenges and what is the procedure adopted to speeden the spying, any inputs will help.
Two things right off the bat you can do to reduce the amount of time it takes to spy all these elements:
Stick to HTML mode like you've said - it'll be faster than any other spy mode in the long run.
Spy a single element of a particular type (say, a file in a list of files) and set the Path attribute dynamically from your object. Enumerate list items and tell Blue Prism which element to run actions on by the same Path attribute.
As an alternative to all this, code against whatever Sharepoint API is available to you. Ditch the UI automation and focus solely on leveraging whatever functionality is necessary using exposed back-end interactions. This eliminates your spying requirement entirely, as everything is handled on a code-level basis. It'll also eliminate a significant portion of your processing time by not having to traverse user interfaces.
I agree with esqew on the point of trying to avoid standard UI automation of SharePoint as much as possible. However, if you are not tech savvy enough to use APIs, an easier alternative is to use a database connection.
This can be achieved easily using Microsoft Access as an intermediary between the list and BluePrism. Here are some guides (video or article)
There are some limitations of course, but if you are interested in reading only then it may be a viable option for you.
The important setting should be done i.e
You might have to do some administrative changes i.e You need to enable Ms Access Engine on SP list
We need to install MS Access engine so that SharePoint List will get compress and just before starting the table you can able to see the Ms - Access logo.
Need to write the macro in MS - Access to fetch the data from SP List and Update/Refresh back to the SharePoint, by doing this you need not dependent importing SharePoint list data manually.
In Ms - Access, all the SP list will be created and stored in table. The macro will just refreshes the table data, you can query MS- Access Extractor file by using OLEDB and also you can use Insert/update/delete queries as well
While opening the Ms - Access File, you just need to launch the SharePoint default site (It is dependent on how coding is done in Ms - Access Macro to access the SP List).
And I would like to say that this is the best and reliable approach to work with SP List as per my knowledge.
Unfortunately, Currently I don't have the MS - Access Code :(. If i could have, I am happy to post it.
Following up on my previous post, I need to be able to query a database of 6M+ rows in the fastest way possible, so that this DB can be effectively used as a "remote" data source for a dynamic Excel report.
Like I said, normally I would store the data I need on a separate (perhaps hidden) worksheet and I would manipulate it through a second "control" sheet. This time, the size (i.e. number of rows) of my database prevents me from doing so (as you all know, excel cannot handle more than 1,4M rows).
The solution my IT guy put in place consists of holding the data on a txt file inside of a network folder. This far, I managed to query this file through ADO (slow but no mantainance needed) or to use it as a source to populate an indexed Access table, which I can then query (faster but requires more mantainance & additional software).
I feel both solutions, although viable, are sub-optimal. Plus it seems to me as all of this is but an unnecessary overcomplication. The txt file is actually an export from SAP BO, which the IT guy has access to through WEBI. Now, can't I just query the BO database through WEBI myself in a "dynamic" kind of way?
What I'm trying to say is, why can't I extract only bits of information at a time, on a need-to-know basis and directly from the primary source, instead of having all of the data transfered in bulk on a secondary/duplicate database?
Is this sort of "dynamic" queries even possible? Or will the "processing" times hinder the success of my approach? I need this whole thing to really feel istantaneuos, as if the data was already there and I'm not actually retrieving it all the times.
And most of all, can I do this through VBA? Unfortunately that's the only thing I will be having access to, I can't do this BO-side.
I'd like to thank you guys in advance for whatever help you can grant me!
Webi (short for Web Intelligence) is a front-end analytical reporting application from Business Objects. Your IT contact apparently has created (or has access to) such a Webi document, which retrieves data through a universe (an abstraction layer) from a database.
One way that you could use the data retrieved by Web Intelligence as a source and dynamically request bits instead of retrieving all information in one go, it to use a feature called BI Web Service. This will make data from Webi available as a web service, which you could then retrieve from within Excel. You can even make this dynamic by adding prompts which would put restrictions on the data retrieved.
Have a look at this page for a quick overview (or Google Web Intelligence BI Web Service for other tutorials).
Another approach could be to use the SDK, though as you're trying to manipulate Web Intelligence, your only language options are .NET or Java, as the Rebean SDK (used to talk to Webi) is not available for COM (i.e. VBA/VBScript/…).
Note: if you're using BusinessObjects BI 4.x, remember that the Rebean SDK is actually deprecated and replaced by a REST SDK. This could make it possible to approach Webi using VBA after all.
That being said, I'm not quite sure if this is the best approach, as you're actually introducing several intermediate layers:
Database (holding the data you want to retrieve)
Universe (semantic abstraction layer)
Web Intelligence
A way to get data out of Webi (manual export, web service, SDK, …)
Excel
Depending on your license and what you're trying to achieve, Xcelsius or Design Studio (BusinessObjects BI 4.x) could also be a viable alternative to the Excel front-end, thereby eliminating layers 3 to 4 (and replacing layer 5). The former's back-end is actually heavily based on Excel (although there's no VBA support). Design Studio allows scripting in JavaScript.
I am having question, we have to handle huge volume of data, like one billion records that should load into a SharePoint list, after loading into the SharePoint list users can edit and update records.
Suggest me best solutions to edit /update one billion records in SharePoint list
Thanks
Ramesh Reddy
1,000,000,000 records? Good luck with that! You may want to revisit the decision to use SharePoint lists as the underlying data store and look at a database instead with some very well constructed indexes.
In case our a masochist though :-
Working with Large Lists in Office SharePoint Server 2007
If its 2010, BCS could be used, and its available in all versions of SharePoint, completely agree with all the others though, you need to rethink your approach, sounds like you need to leverage a proper DB. If you do need to surface the data in SharePoint BCS and External Content Types in 2010 are a god-send.
A SharePoint list is the wrong storage choice here. A quick quote from the Working with Large Lists in Office SharePoint Server 2007 (bottom of the first paragraph under "Test results and findings"):
The maximum number of items supported
in a list with recursive folders is 5
million items.
Perhaps this list is more logically several lists dumped into one?
If you need to expose data through SharePoint, use a regular DB and maybe read about SharePoint Business Data Catalog?
I have a ton of data in a sql database which I would like to be able to import and display in excel (I can already do this) and additionally modify or append to the dataset within excel and write the changes/additions back to the database.
What is the best way to go about doing something like this?
Please let me know, thanks!
The way to do this is via Sql Server's DTS/SSIS capabilities. Create SSIS packages for Excel import and export and execute them as needed.
However you still have the issue of people having to share this massive spread sheet. You should consider importing the data into the db permanently and providing a winforms interface for the data entry. You'd be surprised how quickly you could whip out an app with a databound grid view control that would give you decent, Excel-like ability to add/edit/delete table data.
Although Excel is great at displaying/reporting on data stored within a SQL DB, it has no built-in controls for updating the data.
I would recommend investigating using VBA (Visual Basic for Applications) or based on your coding experience/tools available to you, VSTO (Visual Studio Tools for Office).
This method will allow all of your users to share the spreadsheet at the same time and allow incremental updates plus validation of the data being entered by the user at the point they enter it.
All the usual gotchas apply though - mainly GIGO (Garbage In, Garbage Out). Correctly authenticate your users and what they are allowed to update
I have only been working with sharepoint for three months but right from the start I was told that the SharePoint content db was off limits as MS could change the schema at any time. The recommended route is to use the object model, and in most case I kind of understands that.
Now I need to join some lists in order to present the content grouped by some specific fields. Rather then iterating through each and every list I would prefer to link our own db which resides on the same DB server, to the WSS content DB and just create a view on the tables. This view should be on our DB in order to make such that we don't change ANYTHING on the WSS content DB.
Am I on the route to eternal damnation or not?
Yes, you are. Microsoft is very clear that any modifications to the SharePoint tables renders you unsupportable.
Direct modification of the SharePoint database or its data is not recommended because it puts the environment in an unsupported state.
Now, creating a link on your own DB which queries the SharePoint DB is shaky ground. Personally I'd do one of two things:
If this is a mission-critical application, run it past MSFT support.
If it is anything else, just make sure that your view is not locking the DB during querying.
A better strategy might be to iterate the lists and sync it to your own table so you can do whatever kind of data-mining you'd like - if you don't mind whatever lag time your sync routine would need.
SharePoint pretty much relies on total "ownership" of the underlying database.
Small things like another process reading from the SharePoint database could potentially slow down SharePoint's operations in unexpected ways.
As SharePoint does not usually update in a "real time" manner, it should be good enough to create a process that queries the sharepoint lists and adds the data to a table in your own application.
Schedule the crawl for a low activity period and voila a solution that is not going to risk unexpected slow downs to SharePoint.
Start your search on querying SharePoint at SPQuery.
Check out SLAM, SharePoint List Assocation Manager. It allows you to easily push your SharePoint data to SQL, including complex joins (one to one, one to many, many to many). And it keeps the data synchonized in real time.
http://slam.codeplex.com
Well, if the joins you need to do are pretty simple, defining a linked data source in SharePoint Designer may work for you