Related
This is more general question - I was asked to come up with a document called "SharePoint Strategy", which should descibe (or maybe it's better to say suggest) general company strategy for handling SharePoint in the future. And by handling I mean mainly:
how we manage and support current instance (which is SharePoint Online custom solution provided by external company)
how we evolve current instance with new solutions / features (we have some development skills in the company - 1 dev), so I think the question is how we manage, develop, deploy and support custom solutions
I would be grateful if you can share your experience in this kind of work (if you've had chance to write documents like this or you were part of the discussions). Any idea where I should start?
Regards
George
Your question is not a good fit for this site for "enthusiast programmers", but you raise an important issue that I'd rather try to address than shut down.
What you are after is called a SharePoint Governance Plan.
Nobody can tell your company how your company should be using SharePoint. You will need to work that out amongst yourselves. If you have been asked to come up with a document, you've really drawn the short straw. Because you won't be able to write that document on your own. But you can look at SharePoint governance guidance and sample governance plans and work out from there what decisions your company needs to make, what your company wants to achieve and where your company wants to go.
The Governance Plan helps to document where you are now, where you want to be in the near future, and maybe a few tactical steps to map out how to get there.
The samples and articles you can discover when you google SharePoint Governance Plan sample will only be a starting point. The hard bit comes after you realize that your company needs to make a few decisions, which are above your pay grade, but you may not know who should make them. You will need to collect representatives of the business in a Governance Committee, and they will contribute to the SharePoint road map and the governance plan.
If you work in IT, chances are that you will find it very hard to get the commitment from the business to contribute time and effort to this whole Governance spiel. They just want SharePoint to work and expect the IT department to magically mind-read and automatically apply whatever they need without spending time to detail the requirements.
A SharePoint Governance plan is a little bit like a business plan. It includes vision, goals, strategy and tactics and it cannot be written by one person's efforts alone. It has to be a team exercise.
Try this article for a start: https://sharegate.com/blog/real-world-sharepoint-governance-plan
Edit: If the document is supposed to shed light on difficult decisions, you can present different approaches for each of the scenarios along with pros and cons for the approach.
For example:
Challenge: Developing custom solutions in SharePoint
Option 1: hire external developers
Pros: no need to recruit and/or
train in-house staff
Cons: more expensive, also hard to tell if externals have the knowledge to do the job
Option 2: employ developers
Pros: working full time in-house, easy to manage, cheaper
Cons: hard to recruit, difficult to judge applicant skills in advance
Option 3: do nothing
Pros: don't have to make a decision, phewww!!
Cons: Nothing gets done, arghh!
Again, the point here is that it is not you who has to make the decision. You present options and the business representatives evaluate the options and make a decision, which will then be documented in the next version of the governance plan. The governance plan is not set in stone. It changes all the time. It is a living document about how the business wants to use SharePoint.
Things change. That's a given.
The Governance Plan tries to define how things should be handled with the current information.
The Governance Committee meets regularly to review the Governance Plan and they can change the plan.
The whole idea here is that the Plan comes from what the business wants, not what a poor single soul in IT has been tasked to dream up.
This may not be the most technical question, but I was just interested, nonetheless...
How does a giant company like Google keep from having their code stolen by employees? Maybe I'm wrong, but I would assume that their source code to their search algorithms (amongst other things) would be valuable to their competitors (i.e. Microsoft).
I guess I can best phrase it like this:
What's keeping an unscrupulous
employee who has sufficient clearance from
accessing Google's code repository for
a specific project and copying significant amounts of code
to a flash drive and taking it to their
competitors?
Fear of being sued?
Things within a company like Google are also compartmentalized. So not everybody has access to all code. If someone has access to code, you can bet that Google knows when they access it. I'm sure they have some kind of algorithm that looks and sees if somebody just downloads a lot of files very fast. The search algorithm isn't a small file obviously, it is a gigantic application.
All this would allow them to track who has stolen the code from within. There is also the fact that any self-respecting company or company with something to lose (i.e. Microsoft) would not take anything like this from somebody. They would probably even tell Google about it.
It is called protocol. The idea that only a few people get to know the code. In which then those few have to tell a major very embarrassing secret to the others. So then nobody can tell or else they get outed in the public. Which can be very simple like they like something, compared to as bashful as they are all the way to they killed somebody.
Many employers, including one that I've worked for, completely block flash drives.
In many cases, though, this is to protect non-technical confidential information.
Companies that are serious about protecting their assets will have access logging on their core systems and active scanning to detect suspicious patterns. Similar security is implemented for employees of government agencies (e.g. tax, social security) holding sensitive personal information. Users who access data outside of their assigned cases can be flagged and investigated.
I suspect (but don't know) that similar scanning could be implemented in high value source code repositories.
Some organizations block the use of removable media (It has been reported that some agencies have reacted to Wikileaks with such policies), in some cases by physically gluing up the USB/media ports. This restricts potential thiefs to network transfers of material which can be scanned.
I think companies such as Google will implement access control on their source code repository / version control system. So their employee would only be able to access source code in which they were involved. And their access could be revoked from previous repository if they're being assigned to different project. Its the same thing with normal internal documents, would a security-conscious company let documents be downloaded by any employee freely ?
I think codethis hit the nail on the head. Some fly-by-night operation may be interested, but Microsoft, Yahoo, etc - wouldn't touch stolen code with a ten foot pole. And the fly-by-night wouldn't have the infrastructure. If you didn't tell anybody it was stolen - it's not like you could get away with walking in to a company with an entire spider/searching algorithm on your thumbdrive and declare you wrote it last week.
The bigger threat is details of the search algorithm getting out. SEOers, as a whole, are rather shady - and many would kill for solid facts about how the algorithm ranked or downranked pages. Even then, Google has demonstrated the ability to change their ranking algorithms so quickly that it wouldn't much matter.
On the other hand, Google doesn't have that much super-secret code. Most of their cool stuff (MapReduce et.al) is publicly available (see Hadoop). This question is probably more applicable to a company like Adobe. Some of their Photoshop algorithms are really cool, and would probably hurt them if they got out - but again, no legit company would touch it.
Is there a way of offering the flexibility of Excel/Access development that end users love while instilling centralised IT management so data and logic is secure, backed up, version controlled etc. The common options are to re-write in C#/ASP.Net/Java/Python/Your Choice, but that takes away control from the users. Is there a better way, and what do you do at your site?
There is a universal issue of users creating fantastically useful Excel/Access mini-apps that the IT department would like to bring under control. Users love the flexibility that Excel affords, especially on the fly changes, graphing and data import/export. In Access we have brilliant QBE. The downside is that after a short while there are legions of out of control spreadsheets/mdbs which are mission critical, with lots poorly understood business logic, and brittle code, they're a pain to support especially as staff move on.
This puts the IT dept in an awkward spot, they'd like to support these apps, but don't know enough about them. This is made more difficult as they are typically insecure with zero documentation.
Having been of both sides of the fence I would go after the root cause of the problem. Why do uses make their own little apps? Because it is too hard/expensive/time consuming/never turns out right when they go through the “proper” channels.
The other thing is they tend to know the business very well so whilst their coding might not be very good their knowledge of what needs doing is very good.
So what can we do to combat this problem? I personally think their should be a small team of people within IT whose job (or one of their jobs) is to develop these small applications. They should work very closely with the end users and not be locked in the ivory tower of IT.
In my current role I’m on the non-IT side of the fence, I have a few quite major applications that needed to be developed so I asked for an install of visual studio and some space on an SQL server. I had my request denied. So I just asked for SQL server space, again request denied (each request taking about a week to go through) So in the end I’m “stuck” in access.
Now these are very nice access apps with version control, comments in the (shock!) and all the other nice things but at the end of the day I was trying to do things the “right” way and ended up being forced down the access route. So when my apps try to get scaled up and I’m quoting a long time for a rewrite who is to blame?
Have you considered looking at SharePoint for department-level applications? Many professional developers will balk at the idea of using Sharepoint for "application development," but it truthfully can be a great way for "power users" to start putting their data and tools in a managed framework.
With SharePoint, you can manage the overall structure of the site and then set up users with elevated permissions within their respective departments. There are some great 3rd-party tools to help with keeping an eye on what's going on in your SharePoint site.
SharePoint is not a silver bullet by any means, but it is great for many multi-user applicatinos that need to keep up with a list of data.
(The following is not really related to my above answer, but your question really hit home and I thought I'd share my similar experiences and insights.)
Our company will be going through a similar process in the near future. I'm on the "end user" side of things and can sympathize with a lot of what Kevin Ross said. Sometimes Access and Excel are simply the best tools available for me to get the job done.
Here's an example: I was asked several years ago to come up with a system for creating Purchase Orders to a vendor in China for product for which there is a 3 month lead time. Our ERP software had a few features for procurement, but nothing that even came close to the complexity of the situation we were facing. Years later, after going through several iterations of the application in Excel (VLOOKUP was a lifesaver), Access ("So that is why people using relational databases. Awesome!), and back in Excel ("let's not make this so complicated"), I still find that these Micorosft Office apps are the best tools to get the job done.
What's the cost to not use these tools to get the job done?
Contract work to our ERP vendor to add a special feature for this ordering process: are you kidding me? We'd likely pay tens of thousands of dollars for an unflexible monolithic application with horrendous user experience...and we would still end up back in Excel.
Buy third party software designed for this exact process: I've seen an on-site demo of software that does exactly what I want for our procurement process. It starts at $100,000. There are probably other tools that we can get for a few thousand dollars, but at that price point, I've already emulated most of their features in my own application.
Try to finish the job "by hand." : Ha! I'm a programmer at heart, which means I'm lazy. If it takes a solid week of sitting at a desk to work up a purchase order (it actually did take this long), you can bet I'm going to work up a solution so that it only takes me a few hours (and now it does). Perhaps the guy after me will go back to doing most of it by hand, but I'll use the tools in my toolbox to save myself time and stress.
It's so hard to find the perfect application to allow for maximum creativity on the user end but still allow IT to "manage" it. Once you think you've found a solution for one thing, you realize it doesn't do something else. Can I write I printable report in this solution like I used to do in Access? Can I write complicated Excel formulas that tie multiple data sources together from different sheets ("You want me to learn what? No, I've never heard of a "SQuirreL query" before. VLOOKUP is just fine thankyouvermuch)? Can I e-mail the results to the people in my department? Can it automatically pull data from our back-end database like I do in Excel and Access? Can I write my own code, VBA or otherwise, to make my job easier? The list goes on.
In the end, the best advice I can give to any IT manager in your situation is to respect the other workers at your company. Let them know their work is important (even if it's only useful to them and the guy at the next desk over). Let them know you are not trying to make their job harder. Don't assume they are morons for creating mission-critical applications in office productivity software; they are just trying to get the job done with the tools at hand and are usually quite capable and intelligent people. Invite them to explore different solutions with you instead of just removing the tools they currently have in their toolbox and then replacing them with ones they don't know how to use.
At the end of the day, if you have users who are smart enough to shoot themselves in the foot by creating complicated apps in Excel and Access, they are probably smart enough to learn to use the appropriate tools to accomplish the same tasks. Invest the time and energy to involve them in the process and you will have a solution that works for everyone at the end.
You could try a hybrid approach: Allow your users to use Excel/Access to home-brew their own, specialized tools, but take the mission-critical stuff and put it under IT control. There are a few strategies that could help you with this:
Make sure that your IT department is firm on VBA. Not the "yeah-everybody-can-write-a-few-lines-of-basic" type of knowledge, but in-depth training, just like you would if it were a less simple programming language. Although "real programmers" will tell you otherwise, it is possible to write large, stable applications in VBA.
If you currently have the data in Access databases, move away from that and migrate it to an SQL Server. This allows you to do centralized backup and management, while still giving your power users the flexibility to "link" these SQL Server tables to their Access frontend.
Commonly used business logic should be under control of your IT department. This can be done either with VBA, by creating an Access library that is linked by your users, or in any of the .net languages, using COM interop. The latter sounds more complicated than it is, and it will increase the satisfaction of your IT department, since developing in .net is just much more rewarding than VBA (version control possible, etc.).
I would second one of Kevin Ross's main points:
I personally think their should be a
small team of people within IT whose
job (or one of their jobs) is to
develop these small applications. They
should work very closely with the end
users and not be locked in the ivory
tower of IT.
I think any IT department that has a lot of users using Access/Excel should have at least one properly trained and experienced specialist in developing apps on those platforms. That person would be the go-between to make sure that:
IT's priorities and policies get properly implemented in the home-grown apps.
the end users get expert help in converting their home-grown efforts into something more stable and well-designed.
I would second Tony's point that whoever works with the end users in revising these apps to meet IT standards should work side-by-side with the users. The Access/Excel specialist should be an advocate for the end users, but also for the IT policies that have to be followed.
I also think that an IT department could have a specialist or two on staff, but should also have a full-time professional Access and/or Excel developer as a consultant, since the on-staff people could probably handle day-to-day issues and management of the apps, while the professional consultant could be called in for planning and architecture and for the implementation of more complex feature sets.
But all of that would depend on the size of the organization and the number of apps involved. I don't know that it would be desirable to have someone on salary who is nothing but an Access/Excel specialist, precisely because of the problem you get with all salaried employees compared to consultants -- the employees don't see as wide a variety of situations as an active consultant with the same specialization is likely to see and thus the consultant is going to have broader experience.
Of course, I recognize that many companies do not like to outsource anything, or not something that important. I think that's unwise, but then again, I'm the person that gets hired by the people who decide to do it!
If it's mission critical, and it's in Access or Excel, is built poorly, and no one understands it, it is probably time to rebuild it properly.
When the 'users' are in control it usual means one particular person is in control of the architecture, design, coding and documentation... except they normally omit the documentation step. Source control and bug reporting, the touchstone of software development, is usually absent. Few instances of code reuse, due to the nature of Office apps (code modules usually embedded into documents) and VBA (little OOP, most VBA coders don't use Implements, etc). All this means that the resulting applications are not subject to get proper scrutiny and quality can suffer, meaning there are likely to be maintenace issues, escpecially when that one user leaves. I know because I used to be that person ;)
So in order to satisfy the IT department, the proper process needs to be applied. That one 'power' user can continue to own the design and coding but will get peer review, perhaps the serivces of a technical author and a dedicated tester, be required to use source control, perhaps consider integrating with enterprise systems, etc.
There is no getting around the use of Excel/Access. It's what's available, and still very powerful and flexible. The best thing to do is offer some guidelines as to how files should look and be set up. If everyone is using similar standards then the files will live longer and more productive lives, beyond the creator's tenure at the company.
You've got some excellent answers regarding dealing with the folks and the business side of things. So my response will be more technical.
If you are going to redesign the app have the developers work in the same offices as the users. Given the users updates every day or two. If the users have any minor suggestions give those to the users within a day or two. Ultra Frequent Application Deployment
Give the power users an Access MDB/ACCDB linked to the tables with a bunch of starter queries. Let them create the queries they need to export the data to Excel for their own purposes and distribution to clients.
We've currently got four web servers in a farm generating IIS web logs about 100Mb per day. These can be compressed pretty effieciently down to somewhere around 5% of their size.
We are planning to use waRmZip to move them off the servers and onto a SAN. After a week or so we can be confident we don't have any technical issues to investigate so the only other thing would be using them for trend analysis as a compliment to Google Analytics.
What retention periods do people recommend? Are there any legal requirements to keep this data?
Legal requirements will depend on your country, how much you're logging, and quite possibly the nature of your business. Talk to your company's lawyers - legal advice on SO is likely to be worth what you pay for it.
If you're only storing 5MB per day, you should be able to store them for basically as long as you want without worrying on the technical front.
Please consider the sensitivity of your web log data as well. I have no idea whether access to your web apps would be considered sensitive if made public, but you need to realize that your web logs contain the necessary information to potentially identify individuals (esp. in conjunction with other information available elsewhere). Your privacy policies should reflect how long you retain these logs and what purposes to which they will be put. Google, I think, recently decided to anonymize their logs after 9 months to help protect user privacy. Granted, their situation is a little different since they collect so much information, but you need to consider your customer's needs as well as your own when determining how long and in what form to keep your logs.
I tend to keep mine forever. That's mainly for trend analysis because Google misses some visitors (non-JavaScript ones).
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I didn't know I would be getting too many replies so fast. I can provide more details. It is definitely for use within the company. I was looking for some info on whether I should be more careful or anything to watch out for...
My boss is asking me to put some tracking info on what users do with their application. It is not about collecting sensitive data but there might be some screenshots involved and I am not sure if this is a breach of privacy.
Would love to hear any thoughts on this or if you handled something similar.
At work, there is no privacy. Think of it this way, if you work for a financial institution, or a government one, monitoring users may be the difference between keeping sensitive information secret and not. (I want my personal information kept private). They are paid to do work at work. If they are afraid about what they are doing is wrong, then they shouldn't be doing it.
A comment brought up a good point. If you are selling the product and spying on end users, that is totally different. That is highly unethical to take screen shots and report them back to the company. Actually where I work, we'd have you arrested for it if we found out. (yes, you'd be violating a federal law, and I guarantee we'd go after everyone and sort out the mistakes later.) That is a very slippery slope.
If you mean users at large, yes it's a breach of privacy.
If you mean users internal to your company (workers), then no -- there should be no expectation of privacy in the workplace.
Sometimes it is good to collect some metrics and will help in enhancing the user experience. Once, we were able to prove that a certain functionality was never used and we were able to remove support for it. For screenshots, you should be careful to take only the required window instead of a full screen.
If the application is used internally within your organization, and you have a corporate policy that states "no expectation of privacy" that has been communicated to and signed by your users then there is no issue.
Monitoring the actions of employees within a business in the US is very common practice.
Legal issues aside, do you want to work at a company that takes screenshots of your desktop?
Even if legal, this behavior is sure to drive away developers. Remember, in a bad work environment often the best developers leave first; they have the best job prospects.
Here's a corollary example: would you want your boss taping and listening to phone calls you made from the office? You don't give up every right you have just by cashing a paycheck.
Even if this screen capture methodology is legal, it certainly isn't ethical and will absolutely damage the morale of employees by demonstrating that they cannot be trusted.
It's just a bad idea. There have got to be better ways of accomplishing your goals than this.
Screenshots? If it's not opt-in, I'd say that's a pretty clear breach of privacy.
I made a simple CMS in PHP and I had to store all actions of users, but it's a completely different situation. In my opinion what is asking your boss is a bit out of privacy, especially if in your application you don't mention to the user this kind of behavior.
On a work machine? Absolutely; as long as the users know the extent to which they are being monitored. It's their choice to work for the employer, and they are using the employer's equipment.If you don't notify them that they are being watched, then that is kind of a "grey area"....depending upon state lawss, it may even be illegal - depending on what sort of information you are monitoring.
Something that would help on clarification would be is this an internal company application or something that will be on user's personal computers.
Typically when it comes to computers that are owned by the company, if the company decides to do monitoring, it is their choice. Disclosure of the monitoring is often encouraged in an effort to be open and honest, but is not mandatory. A user should not have any expectation of privacy when using equipment owned and managed by the company.
This is not just a matter of custom built applications, but also web browsing, email, phone conversations, etc. If you are using company resources then you are releasing your privacy.
If this is an application going to users outside of the company, then yes it is wrong without permission by the users.
That is greatly depending on the country you are in and what information you are collecting and what you do with it.
There is a huge difference between the US and EU for instance.
The Law, jurisprudence, union contracts and company policy (when not in contradiction to the above) are what determines what is acceptable.
If its for an internal app its completely ethical.
Beyond disclosing to all users that their use of the apps is monitored there is no other obligation of disclosure(excepting federal contracts and union contracts).
What is most important about capturing this kind of data is to focus on capturing the absolute least amount necessary - capturing screenshots of all open windows plus any adjacent data streams does in fact incur liability issues (think HIPPA) as well as producing a mountain of data that no one will ever look thru until a lawyer requests it with a subpoena and you're asked to go thru it and redact all Names, DOB, and SSNs in 160GB of data.
Seems this has already been answered, but it should be noted that there are countries where this is illegal, even at a place of work.
For instance, in Switzerland it is illegal to track which websites each user has been visiting.
Other than specific laws to the contrary, I would agree that it is acceptable to do, since there should be no reasonable expectation of privacy at the workplace. That said, informing the users is the right thing to do.
One other caveat, if the data you are collecting is sensitive enough that an attacker would have use of it (say, the screenshots include CC numbers), then you must ensure that this information is well protected. (I'm not referring to the user's information, but say the bank's clients' account details.)
If it is done without the user's consent, then it is definitely a breach of privacy. Even with the user's consent, it must be made clear exactly what information is being passed back. If the screenshot was to grab the whole screen, not just a window, then you could potentially get all kinds of private info.
Is this an internal app or a something for the public? If it's internal, it's not unethical, even if it's scummy, to monitor users.
If it's something for the public, in order to not be sleazy:
the user has to be able to opt-out
no personally identifying data can be collected
only data about your app (not screenshots of the entire screen) can be collected
It really depends on exactly what is being collected, the disclosure, and if the program could be opted out of. If that passes the smell test, then ensuring the reporting does not provide an attack vector and the data is appropriately safeguarded becomes your concern. If things seem shady get some written 'feature request' to CYA. The basic idea, if done right, is nothing new. Microsoft, for instance, does it with some of their products.
In a work environment, I think it is OK as long as all employees know that they may be monitored. I've seen places (Intuit was one) where employees are tracked all day. Not my cup of tea, however.
In government facilities, there is typically some sort of login screen that states that anything and everything done on that machine is subject to monitoring.
If these are applications that are run by the general public, I'd say that it better be crystal clear that you are collecting data on them. Personally, I'd rather not have programs 'phoning home' with info about my activities, boring as they may be.
If the client is external, this should be disclosed to the client. Actually, if the client is internal OR external, if you do not disclose it, it is totally unethical.
An employment agreement that states that there can be no expectation of privacy constitutes disclosure.
Screenshots? If it's not opt-in, I'd
say that's a pretty clear breach of
privacy.
you've opted-in by cashing your paycheck :)
as many indicated, informing the user is the best the company can do. Informing, not asking to Opt-In.
I would suggest reading:
Privacy. My interpretation is that people will expect some things to be kept private such as their personal information. By interacting with your sites, users are sharing information with you that you should be able to use but not distribute or abuse as if it was your own.
Screen shots is obviously the hot button issue here. While users entering information into a text input field are knowingly giving you information, screen shots go beyond what a typical user would expect and therefore should be disclosed to the user through a privacy policy.
Collecting anonymous usage should be doable without screenshots.
If your app collects any data that is meant to be protected by privacy laws, then you will have to treat the screenshots as containing sensitive information and protect them accordingly. Data protection laws are pretty strict in most countries.
Unless you have a really really small company, privacy laws vary a lot between countries, and the feature is probably more trouble than it's worth. In any country I've even lived in, that idea would never fly.
But don't ask a bunch of hacks on a site like stack overflow. Seriously, ask a lawyer.
I think the question is still a bit vague as to who is going to be monitored for what. From what I understand who'll be monitored are the end users who are using the application and the gathered data will be used internally. Assuming this is the case, I think, I can contribute the following answer:
If you are going to monitor end users to see how they are using your product, you are in human factors/user experience business and what you want to do is really an experiment. Doing such an experiment requires consent of the subject (the end user). In an academic setting (and I think the same goes for industry as well), there is an Institutional Review Board (IRB) which grants permission for such experiments. I believe in the industry scene there are similar organizations (just not sure what they are called). A request for permission for such an experiment is accompanied by a report which details the user experiment in a very specific manner. The IRB than decides whether to issue a permit or not.
The important point is the consent here and users should know about the experiment and agree to be subjects. I think, in the absence of a user consent the experiment is neither ethical nor legal. Again, I approached this based on an assumption and tried to summarize my experience in such experiments.
Collecting screen shots may be illegal even if employees are notified. This is an issue of local law and federal law. You haven't said which country you are in. In California, for example, monitoring screens might violate both workplace privacy laws and wiretap laws. You should get an opinion of your corporate attorney before implementing this.