We have 5 tables inside a database and we want to sync the data inside those tables to SharePoint online lists. All the modifications will still happen on the database tables, so the sync should only sync New/edited/Deleted data from the database to SharePoint and not from the other side.
The database tables can be accessed using ODBC. So what are the approaches we have to do such a sync:-
Using Power Automate Flow which runs on schedule basis?
Write a .net console application which reads the data from the database and update SharePoint using CSOM?
Other approaches
Any advice?
Thanks
I've been working on a PowerAutomate sync between an Excel Table and a bundle of Sharepoint lists, and one component that is proving quite useful for the Excel -> Sharepoint update direction is the "Sharepoint File or Folder Created or Modified" trigger.
If your database platform has the capacity to create small csv or json files corresponding to the changes you want to make, then one option might be to set aside some "new, change, delete" folders accessible to your PowerAutomate profile and to have your system pass in files with the records to be changed. Particularly if your db tables are particularly large, this might be a more efficient solution than periodically scouring the whole table to try to identify those changes proactively.
Related
There does not seem to be a way to set up automatic refreshing of data from Sharepoint like there is with a SQL server.
The main difference from what I can tell is that Sharepoint is only listed as a Data Table, with no Data Connection Settings.
Data Connection
Data Table
How can I can I set up Spotfire to automatically update the data from Sharepoint once a day? I've tried various JS and IronPython scripts, although not specifically designed for Sharepoint data updates and I've had no luck so far.
I had contacted Spotfire support and at the moment is it not possible to do automatic updating from Sharepoint. They may or may not add it in the future.
I'm hoping this is possible.
The organization I work for has a Sharepoint site and I am able to Upload Files to pages, however I am not an admin on our Sharepoint. I'm not sure what the version is, I think its older (ie: 2005).
I have some Excel Reports I've built. The data for these reports is pulled from a SQL Server Database which I have full control over. I have setup a Job in SQL Server to run every 12 minutes, this procedure pulls in some data and updates a few tables. These tables are used to feed my Excel Reports.
I have a separate Scheduled task set to open my excel report(s) refresh the data connections and save as a PDF.
I would like to link to these PDF Files via our Sharepoint so that the VIPs can access the reports as they want, but they always see the most up to date report.
I was trying to link to a Shortcut to the PDF Files but SharePoint doesn't seem to like that. How do I make the SharePoint link point to the PDF File that is saved over every 15 minutes?
Thanks in advance,
Any insight is greatly appreciated.
The way I do it (newish version of Sharepoint) is make the save location for the PDF the network location where Sharepoint keeps the files for that site. Usually you'll have access to those if you can edit the Sharepoint site.
Here is a tutorial to find that network location.
EDIT: It very well may be disabled by the admin at the moment. But it looks like the functionality is there.
Given the age of your SharePoint (either 03 or 07), most of the modern tools that you could use to do this don't exist for you (Excel reporting, BI tools, etc). The easiest solution I can think of is to actually modify the other side of the equation. A few options:
Change your report to output two copies of the same file. One entitled (as an example) currentreport.xls and the other report20150626.xls . Put the link to the currentreport.xls in SharePoint.
Build an ASP.net page that runs the SQL query you have built and pull the data through a view. Since this would be pulled on demand, it may be a few more cycles of your SQL code, but indexing, caching and selective data pull can prevent this from being an issue. Put the asp.net code in an iFrame in a SharePoint content editor web part.
Build your report using SSRS and host the output of that in SharePoint using an iFrame.
Run a scheduled job in SQL that copies your current report data to a table and query that table instead of your normal report table. That way you only have one Excel file that points to a specific table so no need to update links. You can always keep copying data to specific files if you need a historical record and can't use the DB to store this data for you (though the amount of space that it would take to do so would be minimal).
I have a list in sharepoint which maintains particular month OnCall list,and we are maintaining employee directory in sql server. My requirement is to get complete data from sql server and show it in sharepoint and compare with sharepoint list and show small icon for the employees who are On Call for that particular Month. Can anyone please suggest me the waus of implementing this.
Thanks in advance.
Update: I have finished the part where I have to connect to the sqlserver database and get the employees information. For this we are using 3rd party web part to connect to the sql server and pull the data from the table. Now I have to show some kind of image on the employee name to show that he is on-call for that week. We are going to cretae custom list for maintaing the list of people who are on-Call. Can anyone please advise me on how to accomplish this.
Write a custom webpart which will pull the data from the list using sharepoint object model and SQL server using ADO.NET and do the said comparison.
If you were looking for out of the box, I am afraid there i too little information given here to analyze if its feasible out of the box or not.
If you have the SharePoint Enterprise version, you can look at using the Business Data Catalog. This will let you bind columns to external data sources. This might provide you with the functionality you're looking for.
If you do not have the Enterprise features, do you have access to deploy WSP packages and custom code?
You will have to write your own data access to your external data source. Your options would be to have a job that pulls data from the external data source and populates SharePoint list(s) or create a custom view that pulls the external data on-demand.
You'll have to come up with synchronization strategies. Meaning, is the data in the external SQL data source static, reference information that does not need to be updated depending on what a user does in SharePoint? This seems to be the case based on your question. If you do need to update the external data source, you'll have to hook into the on save event (so probably a custom event handler that listens for ItemAdding) to update the data, validate, and optionally cancel the operation with an error message.
If you can't deploy WSP packages / DLLs, you could take a look at the jQuery SharePoint library. This will let you interact with lists using jQuery. If you also write a WCF or Web Service wrapper around the data you need access to from your external data source that is accessible from the SharePoint environment, you can use hack together a solution.
To accomplish this you'd need to place a Content Editor Web Part on the page you need custom data access. In there you will write the code to reference the jQuery javascript library and jQuery SharePoint library. The code will have to make the calls to your external data service and make any updates you need.
This is the least reliable method to accomplish what you want since it's entirely page-based and can be broken by simply disabling script or someone editing the CEWP or removing it altogether.
If you don't have access to place a CEWP or any of the other solutions, then you have no options at all.
it relatively easy now to pull all the data using the third party webpart and saving it into a custom list. I would recommend you not only creating custom list but also creating the content types for this list. take a look at SharPoint MVP's post about creating a Custom List with Content Types
Just curious what the experience has been in uploading MS Access tables to SharePoint 2007 list. We've been planning on doing so, but I seem to recall issues with SharePoint mangling Access tables in the resulting lists and generally the migrations not going so well. Your experiences? Best practices and recommendations? I'm particularly concerned on its ability to migrate forms and reports as well if it can do so. Thanks!
In response to a question from UK SharePoint User Group I wrote a blog post on Using Access 2007 with SharePoint lists which you might find interesting (some interesting comments too).
There is no facility to migrate forms or reports in Access 2007/SharePoint 2007, only tables. Lots of new stuff in Access 2010/SharePoint 2010 which will allow you to create Access Web databases for SharePoint.
Derek
As mentioned, the big news for Access 2010 is we can build web sites and publish applications to the web. Here is a video of me running ms-access in a browser (the 1st part is in the client, but you clearly see me switch to the browser version of the application about ½ into the video):
www.members.shaw.ca/MrTurtle/2010d1/w1.html
For access 2010 the tables (or so called lists) are going to have things like cascade deletes, cascade delete restrict. There also again a whole bunch of performance improvements that will allow much larger list sizes to perform well.
Note that the new table triggers (data macros) will also go up to SharePoint when you publish the database.
For access 2007, there is not really any particular special problems, but there is just more limitations compared to the next version of access that is coming out. So for example if you need to link up a whole bunch of related tables you have in 2007, when you push the tables up to sharepoint, you don’t have any cascade deletes or any RI options like you’ll have in access 2010.
So I don’t recommend pushing up an application with a whole bunch of interrelated tables. It just not going to work. However, if you have a simple customer list or contact list of a few thousand names that you need to share, then access 2007 should work just fine for you.
So, keep in mind if you have a list or table you need to share, then fine. However, you not going to have any luck or success if you have a whole bunch of tables that are interrelated, and you need to run the application in that fashion.
Edited:
What is the easiest way to scrape extract SharePoint list data to a separate SQL Server table? One condition: you're in a work environment where you don't control the SQL Server behind the SharePoint Server, so you can't just pull from the UserData table.
Is there there any utilities that you can use to schedule a nightly extract?
Is Microsoft planning any improvement here for "SharePoint 4"?
Update Jan 06, 2009:
http://connectionstrings.com/sharepoint
For servers where office is not installed you will need:
this download
There is a SSIS SharePoint task you can use to grab the data info a regular dataflow:
http://www.codeplex.com/SQLSrvIntegrationSrv
Scraping? As in screen scraping? Are you serious? ;)
2 Options
SharePoint Object Model - http://msdn.microsoft.com/en-us/library/ms441339.aspx
SharePoint Web Services - http://msdn.microsoft.com/en-us/library/ms479390.aspx
specifically the Lists web service
The web services is how Excel/Access communicate with SharePoint to integrate with its lists.
In fact a bit of Google foo gives these two results :-
Connecting SQL Reporting Services to a SharePoint List
Accessing SharePoint List Items with SQL Server 2005 Reporting Services
The 2 minute answer is to use Data Synchronisation Studio from Simego ( http://www.simego.com ) just point it at your List and database and it will sync all the changes.
There is an ADO.NET adapter for MOSS 2007/2010 and WSS 3.0/4.0 available which goes under the name Camelot .NET Connector for Microsoft SharePoint. It enables you to query lists in SharePoint through standard SQL language, using SharePoint as a data layer.
Besides from the connector, there will be a large number of open source tools and utilities available, such as webparts for exporting data to various formats (XML, MySQL, ..), Joomla plugins, synchronization services, etc.
See http://www.bendsoft.com for more details and to watch webcasts. BendSoft is currently looking for beta-testers and encourage all feedback from the community.
Example:
SELECT * FROM My Custom SharePoint List
INSERT INTO Calendar (EventDate,EndDate,Title,Location) VALUES ('2010-11-04 08:00:00','2010-11-04 10:00:00','Morning meeting with Leia','Starbucks')
DELETE FROM Corp Images WHERE Image Name = 'marketing.jpg'
I had written a full article about this with step by step screenshot procedures. It does not use any third party components only SQL BI Tools and Sharepoint. Have a look here
http://macaalay.com/2013/11/01/how-to-archive-sharepoint-list-items-to-sql-server/
As Ryan said I would also suggest using object model / web services to store data to separate SQL database. I think that the best approach is to write an event handler that will trigger on your least and copy the data user inserted/updated.
Regarding your query about "SharePoint 4", Bill Gates made some remarks at SharePoint Conference 2008. He suggests enriching SQL tables with SharePoint data, and goes on to mention several other potentially cool things. What exactly he means and whether it will help solve your problem in the future is hard to say until we start seeing betas of WSS4 / MOSS 14.
I would go with the simego software, but i dont have the money, maybe a 15 days trial is enough!
If you have MOSS installed, the Business Data Catalog can be setup from the Sharepoint Central Administration to automagically synchronize data for you. This is a very powerful product and is included with MOSS. I love it when a client has it enabled so I can take advantage of it.
But some don't and for myself, I've found that if they don't have BDC running and available, inevitably they don't give developers many rights to SQL Server so SSIS is generally out of the question (but maybe that's just me). No problem; for those I'll pull together a lightweight EXE that runs on a scheduled task that queries Lists.asmx and pushes changes to a SQL Server table. Fairly trivial stuff for a simple list where nothing is deleted. Get yourself Visual Studio 2008, CAML Builder, and prepare for a good time. The Lists.asmx results is a little funny in that a list's row's fields are each a single node with a lot of attributes, with no child nodes ... something like this off the top of my head ... just remember that when coding ...
<z:row ows_Id="1" ows_Field1="A1" ows_Field2="B1"/>
<z:row ows_Id="1" ows_Field1="A2" ows_Field2="B2"/>
Complications in code occur with copying lists where items are deleted, or where there is a parent/child relationship between SP lists. You'd think I'd have some code to send you, but I haven't bothered putting together something I could reuse.
I'm sure there's other ways of handling it, but the scheduled task EXE so far has been reliable for me for multiple apps for multiple years.
i wrote some code to achieve it, you can find it over here
extract data from moss 2007
Depending on the exact nature of the data you need to insert, it may be possible to just use the auto generated RSS feed to get the information you want, a process will need to read the rss and formulate a query.
Otherwise a consoleapp/service could use the object model to do the same thing, but with more control over field information.
I wish something like this was much easier to do. Something that didn't need SSIS and was boiled down to a console tool that reads a xml config file for source/target/map info.
http://blogs.officezealot.com/mtblog/archive/2008/06/03/importing-list-data-into-sql.aspx