I'm looking at designing some core information systems at a new company I'm working at (described one of my ideas here Workflow system)
I've thought a bit more, and am strongly considering using sharepoint for a lot of the heavy lifting seeing as it comes with so much out of the box.
However, I'm not sure how it will handle the high volume of data we'll be throwing at it. I read the MS whitepaper (http://go.microsoft.com/fwlink/?LinkId=95450&clcid=0x409), and it says about 2000 items in a list is about the limit using traditional design methods.
But first a bit of info on my plan and data structures :
We have multiple clients. Each client has multiple applications. Each application will have multiple, ongoing jobs (or process runs).
Each application will store significant correspondence and documentation. Each job represents the processing of a data file on a single run, and stores information about the job such as the postscript file, postal manifests, etc.
Job volume will be about 50 - 100 a day. Each job will have a workflow, triggered by external programs. Then, say on a "job scheduler" page, production staff can schedule the jobs and perform custom actions on the job (written as plugins).
I was thinking the jobs would sit outside and accessed via the BDC, but I would still like them represented in sharepoint lists, to add in sharepoint functionality and reporting, and they'd be accessible in multiple places
e.g.
Application portal - see jobs for application
Production scheduler - see lists of upcoming jobs, assign to resources, trigger other functionality (e.g. copy print file to printer, produce mailing machine file)
Invoicing view - view completed but uninvoiced jobs, export to accounting package
Client view - client portal displays jobs, invoices, stock levels (from external warehouse system), documentation, change register / helpdesk
So basic info about the job would sit in the BDC, but then sharepoint would capture additional metadata about each job. Also, down the line we might put in more advanced workflows using WF or something like K2 blackpoint / blackpearl.
Is this feasible? Any resources you'd recommend to read to get up to speed?
To use SharePoint, you should concentrate on what SharePoint is good at and what it is designed for.
SharePoint is a great collaboration portal, it is not so good as a simple high volume database. So...
You can setup a small site for each client and subsites for each job. The goal of the "job site" is to display (using a webpart perhaps) the relevant upcoming jobs, a list of job errors/exceptions and relevant team documentation on each job.
Separate sites can be created to give a particular "view" of the jobs. E.g an "Invoicing" site can be created to give a view again from BDC webparts of what is requiring invoicing.
https://iwsolve.partners.extranet.microsoft.com/SDPS/ may provide some help.
Don't try and store huge amounts of information in a SharePoint list, just because it may be possible to "tag" it with meta data. A database table is perfectly able to include columns supplying additional information if required.
Think about it this way. If you are creating 50-100 jobs a day, putting that data into a list pre-supposes your sites users are going to want to enter metadata on those jobs manually. I thought not, so create systems you need in order to get the metadata stored correctly at source, or store metadata about the "types" of jobs within a SharePoint list and allow SharePoint to match the job type with jobs in the BDC.
SharePoint will help you to integrate all your systems information together, but unfortunately it looks like you have a lot of work to do just planning what information should go where and how each type of use will view it.
Please take a look at this blog post I wrote on managing large SharePoint lists for better performance- it might offer a bit of an explanation for the 2,000 items issue, which is not actually a hard limit on the number of items in a list, as SharePoint will support up to 5 million items per list. One way around this would be to create and maintain different views that filter by an indexed field to show you different items, up to 2,000 at a time. Hope that helps.
Dina Ayoub
Program Manager
Windows SharePoint Services
SharePoint is probably quite a good fit for the UI side of things, though you'll need to think carefully about which parts are stored and modified in SharePoint lists and which parts are stored elsewhere. That's not so much a SharePoint issue as something you always have to deal with when you have multiple data sources.
I'd probably use a SharePoint list as the primary store for jobs, to avoid any sync issues and make editing easier. The volume of data shouldn't be an issue - just make sure you aren't trying to display 2000 items at once - it's the view, not the list itself that runs into performance issues on large numbers of items.
Tough question Dane... I would like to know a little more about your design / vision before giving an opinion.
Based on what I read in your question I would not use SharePoint 2007 as a development platform for this application.
1) Development experience in SharePoint 2007 can be painful and unproductive at times.
Hard to debug
Steep learning curve
2) Easy to get in trouble with performance
Data Layer is complex and can require expert SQL / SharePoint Admin skills to make platform scale.
Content databases should not exceed 100 GB.
3) Deployment can be extremely difficult depending on what you are doing.
4) New version will be released in the next 12 months.
Just my .02.
Related
We're starting to use SharePoint 2013 to manage our department's process documentation and I have some questions about best practices for site structure. I'm a little surprised I can't find the answers via web search, since this seems like a basic question every new SharePoint user must deal with.
Moving from a file share environment, I'm trying hard to get out of that mindset and I understand the many benefits of SharePoint over file shares. I also understand why creating folders in SharePoint forces arbitrary divisions on files whereas one large set of documents with metadata lets you filter and group the files based on different needs.
What's confusing me is that I also read that it's better to have too many sub-sites than not enough. It seems like sub-sites can easily become pseudo-folders and I'm not sure where that line is crossed.
Here's an example.
We have a SharePoint site devoted to our department. We've create a sub-site dedicated to an application we developed to load data into our business systems. It mainly holds technical documentation about the application. This application supports many different data sources, each with its own set of user instructions for loading, its own schedule (calendar), contact lists, supporting files, etc. There's no compelling reason to separate them to restrict access. However, there doesn't seem like a lot of value in having them all in the same sub-site, either, since someone working on a job will only want to see the docs and supporting files for that data source. I just can't foresee someone wanting to view supporting files across all data sources, although, I could see someone wanting to see the schedule for all data sources combined.
My question is, should I create separate sub-sites under the application for each data source or do I just store everything in the application sub-site and use metadata and views to group things by data source? Putting all the items for a specific data source into its own sub-site seems to be much simpler to manage and present than having to specify metadata for every new file and creating a lot of views. However, I can't shake the feeling that I'm still using file share thinking. Or maybe I'm just missing some basic concept of SharePoint.
Any advice or links to good discussions of this topic would be greatly appreciated. Thanks.
I would recommend that you use metadata and views to separate data in one repository/site.
My reasons are as below:
In SharePoint, it is recommended to use metadata than "evil"
folders(or subsites in your case). Keep in mind that maintaining
multiple subsites requires big administrative efforts in long term,
for example, some sites will be inherited while others unique
permission.
As time passes by and people rotate, it becomes vague
that where the data was stored and where the new data should go to,
especially with large volume subsites.
Since confidentiality is not concerned in your case, keep data centralized and open to people working in related field increases sharing and collaborating phenomenon. In contrast, using subsites increases the possibility of data silos.
people are all lazy :). Taken me as example, I dont want to remember all those xyz URLs, I want to go to one place and be able to fetch everything that I need.
I'm the IT Manager at a mid-size manufacturing company. We are getting our feet wet with SharePoint - so far we're got one blog in production use> It's the CEO's.
We have use cases for a couple of list-based "applications" with some simple workflow that will be implemented by one of our developers. We also want to give our users (at least the more tech-savvy ones) the ability to create and work with their own departmental sites.
We're concerned, however, that we might be starting something that could quickly get out of control if it's widely adopted (which would be a good thing). Since we don't really understand all the architectural trade-offs, we could end up with massive amounts of user data in a structure that bites us down the road.
Our biggest question is whether to have multiple sites for each use vs. a single root site from which everything else descends. Multiple sites would give us flexibility to make changes or develop new features without creating problems for all the users. However, multiple sites might be harder to back-up, search, and maintain user profiles/security. A single massive site seems to reverse the cost/benefits.
I'd appreciate any insight on the one vs. many trade-offs, or links to resources that discuss it. Links to general SharePoint "enterprise best practices" (sorry) would also be appreciated.
Thanks.
However, multiple sites might be harder
to back-up, search, and maintain user
profiles/security. A single massive
site seems to reverse the
cost/benefits.
I would consider this as incorrect. First we need to clarify when we say multiple sites, do we mean multiple site collections or multiple sites - they are two entirely different things.
Now even if they are multiple different site collections, in SQL database, they are just one database, since the database is created as web application level and not site level.
That was regarding backup.
Coming to search and user profiles, again your assumption is wrong. Search and User Profiles are Shared SErvices and they work fine as long as they reside in single Shared Services Provider. Both are farm level services.
A single massive site is (if you really mean site here not site collection) is a complete no-no and a bad design.
I would recommend having multiple site collections (something like Overall department in your company like HR, Finance , IT) and then have subistes under it. This way you have one database in SQL to manage and still you can scale by adding content database to existing web application.
Again here, I assume that you are creating your topology at company level. If this is at some lower level it needs to be refined.
Read some articles on taxonomy and site architecture on Technet before going ahead with any one.
Planning worksheets for SharePoint Server 2010
http://technet.microsoft.com/en-us/library/cc262451.aspx
Plan sites and site collections
http://technet.microsoft.com/en-us/library/cc263267.aspx
Sites and site collections overview
http://technet.microsoft.com/en-us/library/cc262410.aspx
Plan site navigation
http://technet.microsoft.com/en-us/library/cc262951.aspx
It purely depends upon your needs and requirements. even having a deferent web applications for deferent site i can provide you one citation taking backup as advantage. You might have few sites where data does not changes frequently like organizational policies, process documents etc. in this case taking regular backups/search crawling does not make sense(although you can opt for differential backup and incremental crawl but still in a week or fortnightly you have to take full backup). hence i would suggest carefully analyze your requirements and then take a decision. Microsoft has provided a good list of checklist and templates for planning purpose. few of the links are provided in madhur's reply and rest you can google upon.
I'm currently looking at developing an application, and have a choice between doing a standard ASP.NET web application or integrating it into SharePoint. Our client would like it to be SharePoint if possible, as they are under pressure to put all new development into it, but standard ASP.NET is still an option.
It is an application to manage and view data in a database with about 10 tables, including an approval workflow when certain new items are added. Referential integrity of the data is important.
I have experience of developing ASP.NET applications, but very little with SharePoint. Does anyone have any criteria they would apply to deciding between the two?
So far I am thinking along the lines of:
Referential integrity of the data is important and SP doesn't seem to handle this very well without writing lots of custom code
SharePoint doesn't seem very scaleable with the suggested limit of 2000 items in a list
The application has an approval workflow, which does seem to be something SP does well
On the wholem, it seems like we would end up writing lots of custom code and not really using any of the out of the box SP features. So my thinking is why not just write a standard ASP.NET application.
Are there any other key things we should consider?
By now, you may have already found this link: http://blogs.msdn.com/sanjaynarang/archive/2009/06/19/should-i-build-my-application-in-sharepoint-vs-asp-net.aspx. If not, it's a decent starting place with some good questions to ask.
What follows is my take as a long-time .NET developer (for as long as the platform has been around) and a SharePoint architect (since 2003). That's basically my way of saying that I've been on both sides of the fence.
In my opinion, SharePoint is a platform, not a product. As ASP.NET provides valuable web-based services to the core .NET framework, SharePoint supplies additional services and capabilities on top of ASP.NET. The platform removes the need for writing common code pieces that are a part of so many ASP.NET apps: security code, user profile management, personalization, UI/UX baseline, etc. When you get into the plumbing, you get even more: rich caching support that requires minimal configuration, customization modularity via Features, and more.
Should every application be built in SharePoint? I'd never push for that. With my current client, we use a mix of SharePoint-based and custom ASP.NET applications. Whether an application gets built in SharePoint vs. written from the ground-up in ASP.NET is a function of what we're doing. We conduct the same sort of exercise you are. If SharePoint's features and functionality can be brought to bear to reduce development time, it goes in SharePoint. If the need is too specific or we feel we'd be working around SharePoint, we go the custom app route.
You have some very specific concerns for your application, so let me take a crack at them with the little I understand about your requirements:
REFERENTIAL INTEGRITY: based on what you're saying, it sounds like your data model is pretty specific. Building your information architecture to natively leverage site columns, content types, and lists probably doesn't make sense. That doesn't throw SharePoint out, though. There's absolutely no reason why you couldn't build the data model you want (presumably in SQL Server) and then consume it with components that reside in SharePoint. If you're using MOSS, some of the BDC WebParts might work for you straightaway. If not, you'd still be writing controls and/or pages to work with the data, correct? There's nothing wrong with using SharePoint as your presentation layer for access to SQL directly or (in a more scalable, n-tier fashion) go against business services somewhere else.
2000 ITEM LIMIT: this is a common concern and one that is misunderstood. There is no 2000 list item limit; the actual measurement is 2000 items per view (and that's with out of the box views, by the way) or "container" (such as a folder). You can store many more times that (millions, if you like) in a list if you partition with folders, build your own view to page, etc. Again, given your data structure and the likely need you have to dodge SharePoint's lists, this wouldn't be an issue if you simply consumed data from SQL Server.
WORKFLOW: SharePoint is nice as a workflow host, and the OOTB workflows are handy. I'm assuming you're looking at MOSS (versus straight WSS), but just in case: the approval workflow comes with MOSS. If you're constrained to WSS, you only have one workflow available to you: the three-state workflow.
At the end of the day, SharePoint is .NET and built on top of ASP.NET. Much of the code you'd have to write in a SharePoint app you'd need to write in a custom .NET anyway. I'd look at things from the perspective of understanding whether the experience and features SharePoint affords you (as a developer) can help speed your development cycle and/or improve the user experience (something we, as developers, sometimes forget).
David in Dakota does have an excellent point, though, in that the dev experience for SharePoint is different from straight ASP.NET. The need (or rather, best practice) to deploy via Features, understand specific SharePoint concerns (e.g., lifetime and disposal of SharePoint objects), etc., mean that there will be ramp-up time if you do build in SharePoint. There are quite a few good resources out there (including folks here on StackOverflow) that can help, but you'll need to factor some learning into the equation of whether or not SharePoint makes sense.
One more parting thought: Microsoft is slowly shifting many of its own products and platforms to leverage SharePoint as their UI/UX layer, and the trend is picking up some steam. PerformancePoint, Project Server, Team Foundation Server, and Commerce Server all use SharePoint as their presentation tier. The trend will probably continue, though I don't know how far. If you use any of these products (or their on your technology roadmap), a SharePoint investment now might pay off later.
Despite all of my writing about and advocating for SharePoint, I don't think it's the right tool in every scenario. I still build WinForms apps, smart clients, command line apps, and more quite a bit. It just comes down to weighing "what I get" for "what I spend" (in both time and money).
I hope this helps!
Your evaluation is quite accurate. (It would help to have more detail about every feature your application needs but that's not really practical for this medium.)
The problems you mention have been largely solved but you would need to understand and implement the solutions. For example, there is a CodePlex project that can assist with referential integrity and there are recommendations on how to manage the number of items in a list. But using SharePoint is never going to give you the freedom of writing an ASP.NET application from scratch.
Another thing to consider is how you and/or the client expect the application will evolve in the future. If it will need more collaboration-style features or features such as version history on list items and integration with the Office client then SharePoint may be the better option.
You should also think about the complexity of deploying and updating your application on SharePoint.
I have the following situation:
MOSS 2007 Server Environment A -> Intranet
MOSS 2007 Server Environment B -> Collaboration Environment (approx. 150 site collections for various issues)
Both environments are on different infrastructures but we use the same Active Directory and the same groups. Now we would like to implement the following 2 things:
An overview page within the intranet with all available site collections on environment b.
An overview page within the intranet with only those site collections the user has access on.
now i'm searching for some good ideas what would be the best way to realise something like this.
thanks in advance for any response.
The main thing to be careful of in a solution like this is performance, particularly for your second requirement. That would require looping through every site collection and retrieving permission data, either using the web services or the object model.
I would recommend writing a custom timer job (or two for each requirement if that makes more sense) to execute at a low-traffic time and aggregate this information for storage in a custom SQL database. If there is never low traffic then delay your requests to reduce impact on the server.
A custom web part (or again, two if more appropriate) can then be deployed to both environments. The web part would query the database for the required information and display it to the user.
If the timer job needs to update this data more frequently then you would need to implement some sort of in-memory caching. Depending on your requirements this may need a lot of memory.
For the better part of 10 years + we have relied on various network mapped drives to allow file sharing. One drive letter for sharing files between teams, a seperate file share for the entire organization, a third for personal use etc. I would like to move away from this and am trying to decide if an ECM/Sharepoint type solution, or home grown app, is worth the cost and the way to go? Or if we should simply remain relying on login scripts/mapped drives for file sharing due to its relative simplicity? Does anyone have any exeperience within their own organization or thoughts on this?
Thanks.
SharePoint is very good at document sharing.
Documents generally follow a process for approval, have permissions, live in clusters... and these things lend themselves well to SharePoints document libraries.
However there are somethings that don't lend themselves well to living inside SharePoint... do you have a virtual hard drive (.vhd) file that you want to share with a workmate? Not such a good idea to try and put a 20GB file into SharePoint.
SharePoint can handle large files, and so can SQL Server behind it... but do you want your SQL Server bandwidth being saturated by such large files? Do you want your backup of SQL Server to hold copies of such large files multiple times?
I believe that there are a few Microsoft partners who offer the ability to disassociate file blobs from the SharePoint database, so that SharePoint can hold the metadata and a file system holds the actual files, and SharePoint simply becomes the gateway to manage access, permissions, and offer a centralised interface to files throughout an organisation. This would offer you the best of both worlds.
Right now though, I consider SharePoint ideal for documents, and I keep large files (that are not document centric) on Windows file shares.
Definetely, use a tool.
The main benefit here is version control. Being able to jump easily to a previous version, diff'ing and seeing who modified what (see most VCS' blame/annotate tool- it prints out a text file showing when/who modified each line in the text file).
Second, you can probably benefit from issue tracking/task tracking.
Other benefits include web access from the internet, having a wiki (which can be great in some situations), etc.
I use Subversion + Redmine at work, and I find it highly useful- test a few solutions and you will surely find out further advantages for you.
One thing that can be overlooked in the change to an document management tool is the planning required around how much is going to be stored and information architecture issues like where different content is going to end up.
SharePoint particularly is easy to setup without a good plan going forward and is particularly vulnerable to difficulties later on when things get to busy.
I would not recommend a home grown app for something like this. The problem has been solved by off the shelf tools and growing one from scratch is going to cost a huge amount and not get you any way near the features for the money.
Did I mention how important planning your security groups and document areas (IA) was?
If you need just document storage then sharepoint can do very well. WSS is ewen free and it provides very good document storage capabilities.
But you have to plan carefully as updating existing applications is painfull. If you decide to go with Sharepoint then I can give you few advices from top of my head
Pay attention to security configuration (user groups, privilegies,..)
Plan your document libraries well as it is not easy to just move documents betveen them
Also consider limiting number of versions that one document can have, because sharepoint stores full backups betveen verions, not just changes
Don't use infopath:) we have very bad experience with it (just don't tell this to managers)
If you don't really need to change graphical look of Sharepoint than don't bother with it as it brings many problems (I'm talking about custom masterpages and custom site templates)
Try to use as much OOB stuff as possible, because developing your own webparts not only cost more, but it can be quite complicated.
Make sure to turn-on search indexing. This is quite tricky, because it is by default turned off and then you will be as surprised that search is not working as I was :)
If you try to just deploy it and load 10.000 documents into it then you will surely have problems with it later. If you give a little thought about structure then you will end up with really good document storage.
Migrating is very probably worth the cost in the long term. You will gain reliability, versioning, traceability, and extensibility.
Be sure to first identify the groups/rights, and to identify which links need to be fixed (maybe you have applications that use links to the shares).
An open source alternative to SharePoint is Alfresco, it is very good for CIFS (Windows shares) too.