Basically, my question is to get a feel for what time of day you do things like DNS transfers when moving sites across servers, and provide updates to sites. Does anyone do this during the day, or do most developers do this at night?
I think different approaches can be valid, depending on your situation, the actual change to be performed, how often a change happens and what your users expect.
If you do it during the day, you potentially have a longer time to fix issues if there are any, including if they are issues that need to be dealt with your service provider (which might be more responsive during business hours than at night or during week-ends).
If you enjoy working at night or during week-ends and you don't rely on any service provider to solve any issue related to the move/migration you're asking about, doing it at this time might be less disruptive to your users.
But all that also depends on who are your actual users. More and more users are spread throughout the globe and can interact with your sites at any time of day or night.
Common systems administration practice, when dealing with big changes or migrations, usually recommends not doing it just before off-time (be it night, or week-end, or holidays), unless you're sure that you (or the team) is available to work to fix issues. Basically, you need to plan for issues, because they always happen.
Something else to consider is any SLA you, or your company, might have with the users. In that case, you might need to perform your maintenance action outside of business hours, or with provisional notice, or risking any penalty to be applied.
A proper answer to the question you're asking really depends on your situation and the business you're in.
Related
I've come across many clients who aren't really able to provide real production data about a website's peak usage. I often do not get peak pageviews per hour, etc.
In these circumstances, besides just guessing or going with what "feels right" (i.e. making it all up), how exactly does one come up with a realistic workload model with an appropriate # of virtual users and a good pacing value?
I use Loadrunner for my performance/load testing.
Ask for the logs for a month.
Find the stats for session duration, then count the number of distinct IP's blocked by session duration.
Once you have the high volume hour, count the number of page instances. Business processes will typically have a termination page which is distinct and allows you to understand how many times a particular action takes place, such as request new password, update profile, business process 1, etc...
With this you will have a measurement of users and actions. You will want your stakeholder to take ownership of this data. As Quality assurance, we should not own both the requirement and the test against it. We should own one, but not both. If your client will not own the requirement, cascading it down to rest of the organization, assume you will be left out in the cold with a result they do not like....i.e., defects that need to be addressed before deployment to production.
Now comes your largest challenge, which is something that needs to be fixed with a process issue by your client.....You are about to test using requirements that no other part of the organization, architecture, development, platform engineering, had when they built the solution. Even if your requirements are a perfect recovery, plus some amount for growth, any defects you find will be challenged aggressively.
Your test will not match any assumptions or requirements used by any other portion of the organization.
And, in a sense, these other orgs will be correct in aggressively challenging your results. It really isn't fair to hold their designed solution to a set of requirements which were not in place when they made decisions which impacted scalability and response times for the system. You would be wise to call this out with your clients before the first execution of any performance test.
You can buy yourself some time. If the client does have a demand for a particular response time, such as an adoption of the Google RAIL model, then you can implement a gate before accepting any code for multi-user performance testing that the code SHALL BE compliant for a single user. It is not going to get any faster for two or more users. Implementing this hard gate will solve about 80% of your performance issues, for the changes required to bring code into compliance for a single user most often will have benefits on the multi-user front.
You can buy yourself some time in a second way as well. Take a look at their current site using tools such as Google Lighthouse and GTMetrix. Most of us are creatures of habit, that includes architect, developers, and ops personnel. We design, build, deploy to patterns we know and are comfortable with....usually the same ones over and over again until we are forced to make a change. It is highly likely that the performance antipatterns pulled from Lighthouse and GTMetrix will be carried forward into a future release unless they are called out for mitigation. Begin citing defects directly off of these tools before you even run a performance test. You will need management support, but you might consider not even accepting a build for multi-user performance testing until GTMetrix scores at least a B across the board and Lighthouse a score of 90 or better.
This should leave edge cases when you do get to multi-user performance testing, such as too early allocation of a resource, holding onto resources too long, too large of a resource allocation, hitting something too often, lock contention on a shared resource. An architectural review might pick up on these, where someone might say, "we are pre-allocating this because.....," or "Marketing says we need to hold the cart for 30 minutes before de-allocation," or "...." Well, you get the idea.
Don't forget to have the database profiler running while functional testing is going on. You are likely to pick up a few missing indexes or high cost queries here which should be addressed before multi-user performance testing as well.
You are probably wondering why am I pointing out all of these things before your performance test takes place. Darnit, you are hired to engage in a performance test! The test you are about to conduct is very high risk politically. Even if it finds something ugly, because the other parts of the organization did not benefit from the requirements, the result is likely to be rejected until the issue shows up in production. By shifting the focus to objective measures even before you need to run two users in anger together there are many avenues to finding and fixing performance issues which are far less politically volatile. Food for thought.
We are looking to implement an open source identity management system and have identified ForgeRock's stack as the best technology to implement.
The high cost of ForgeRock support and its per-User pricing model, however, is a potential roadblock. Our current User base is ~45K, but we expect to ramp up to 1M in the next 2 years.
So we're looking into scenarios where we proceed without FR Support. The lack of FR Maintenance releases would seem to put a damper on that, so we're curious if others have gone that route.
What has been your experience?
What kind of projects have you done this for? Size, etc.
In the absence of FR's Maintenance releases, have you been able to easily create your own patches?
What are some potential pitfalls?
If there are blogs or other communities that deal with this topic, please point me in their general direction.
Thanks.
As a community user I did use OpenAM(/OpenSSO) and OpenDJ for the past 6 years or so, but it was a very small deployment (10k users only 1 server instance from both products).
1) In the early stages we did have reliability issues with OpenAM, which we mostly resolved by restarting the server instances - clearly wasn't preferred, but we didn't really spend too much development effort on actually trying to resolve it (plus lacked the necessary knowledge for investigation back then). After spending some actual effort on trying to learn the product it turned out that the most of our issues were either self-inflicted (badly written customizations, or misconfigurations), or was actually something that got recently resolved in the OpenAM project and was relatively simple to backport to our version.
Of course the experience itself largely depends on how often you want to make configuration changes in the deployment though, since we weren't changing a lot of things over the years, OpenAM just worked nicely for long intervals without requiring any kind of maintenance.
3) Since we didn't really ran into new issues (the config barely changed), there weren't too many surprises after a while. The security patches were mostly simple to backport and didn't cause too much trouble (It did help that after 1,5 years I became a FR employee and I actively worked on OpenAM issues though :) )
4) I think running without subscription has its risks, but they mostly relate to:
are you planning to roll out new features based on OpenAM functionality during that 2 years (i.e. are you planning to constantly make changes to the deployment)?
do you have good developers to work on these features? Working with OpenAM for example can quite easily require you to have a look at the source code to figure out how things work, the quality of the documentation has improved a lot over the years though. Regardless, backporting fixes are going to be more and more difficult over time, as the releases will differ a lot more (since the development team is getting bigger and bigger for each projects) - and even then you can't just assume that all the issues you run into are by definition already resolved in trunk. The need to resolve some of the issues on your own is a cost/risk you need to take into account.
what kind of SLA do you want to have for your deployment? Is your business going bankrupt after a 1 minute outage? Is it acceptable to just frequently restart your service (in case you run into some weird issues)?
do you really need support for all 3 products? For example my background would allow me to work easily without OpenAM support, but I would be in the deep end if something is going wrong with my provisioning system...
And a generic remark:
Having user growth of 20x within two years sounds a bit unrealistic, or very hopeful at least. Maybe what you should look for is a 1 year subscription for a bit more reasonable target number and then have a renewal once you have a better understanding of customer growth in your business?
I have been search about SharePoint Governance for past few days, the more I search, more confused I am getting about this Topic.
Could anyone just explain in brief? What you know about it or are you using/implementing it?
This is one of those topics that tend to mean something a bit different when you get to the details for different companies, but basically here's my experience with it.
When you talk about governance its about who is responsible for different elements of it, from defining and managing the taxonomy, to the content for different areas. What the process of publication is, for modification is. Who administers different sections, how documents are to be handled/managed/archived/retained, what the auditing policy is. How escalations will be managed, who needs to be notified, who the stakeholders are, etc.
Basically in other words it's about looking at SharePoint as being not just a content storage system but needing to be a coherent enterprise tool with all of the key stakeholders sitting up and taking notice, and recognizing that some things need clear process to avoid either redoing everything in 18 months or being unable to find things.
Some of the most important things that I find ends up getting discussed is security (internal vs. external access, employees vs. contractors vs. external people, logging of who accessed files, etc.), roll back & geographic distribution (for performance), and revocation of rights (when someone is fired, how is their access revocation managed quickly and effectively).
Sorry this is a bit of a shotgun answer, but that's the type of stuff you're looking at. There are actually consultants who have specialized in these areas. It's usually (from what I experience) about 20% technical, 40% change management and 40% process / business.
I hope that helps.
Is there a way of offering the flexibility of Excel/Access development that end users love while instilling centralised IT management so data and logic is secure, backed up, version controlled etc. The common options are to re-write in C#/ASP.Net/Java/Python/Your Choice, but that takes away control from the users. Is there a better way, and what do you do at your site?
There is a universal issue of users creating fantastically useful Excel/Access mini-apps that the IT department would like to bring under control. Users love the flexibility that Excel affords, especially on the fly changes, graphing and data import/export. In Access we have brilliant QBE. The downside is that after a short while there are legions of out of control spreadsheets/mdbs which are mission critical, with lots poorly understood business logic, and brittle code, they're a pain to support especially as staff move on.
This puts the IT dept in an awkward spot, they'd like to support these apps, but don't know enough about them. This is made more difficult as they are typically insecure with zero documentation.
Having been of both sides of the fence I would go after the root cause of the problem. Why do uses make their own little apps? Because it is too hard/expensive/time consuming/never turns out right when they go through the “proper” channels.
The other thing is they tend to know the business very well so whilst their coding might not be very good their knowledge of what needs doing is very good.
So what can we do to combat this problem? I personally think their should be a small team of people within IT whose job (or one of their jobs) is to develop these small applications. They should work very closely with the end users and not be locked in the ivory tower of IT.
In my current role I’m on the non-IT side of the fence, I have a few quite major applications that needed to be developed so I asked for an install of visual studio and some space on an SQL server. I had my request denied. So I just asked for SQL server space, again request denied (each request taking about a week to go through) So in the end I’m “stuck” in access.
Now these are very nice access apps with version control, comments in the (shock!) and all the other nice things but at the end of the day I was trying to do things the “right” way and ended up being forced down the access route. So when my apps try to get scaled up and I’m quoting a long time for a rewrite who is to blame?
Have you considered looking at SharePoint for department-level applications? Many professional developers will balk at the idea of using Sharepoint for "application development," but it truthfully can be a great way for "power users" to start putting their data and tools in a managed framework.
With SharePoint, you can manage the overall structure of the site and then set up users with elevated permissions within their respective departments. There are some great 3rd-party tools to help with keeping an eye on what's going on in your SharePoint site.
SharePoint is not a silver bullet by any means, but it is great for many multi-user applicatinos that need to keep up with a list of data.
(The following is not really related to my above answer, but your question really hit home and I thought I'd share my similar experiences and insights.)
Our company will be going through a similar process in the near future. I'm on the "end user" side of things and can sympathize with a lot of what Kevin Ross said. Sometimes Access and Excel are simply the best tools available for me to get the job done.
Here's an example: I was asked several years ago to come up with a system for creating Purchase Orders to a vendor in China for product for which there is a 3 month lead time. Our ERP software had a few features for procurement, but nothing that even came close to the complexity of the situation we were facing. Years later, after going through several iterations of the application in Excel (VLOOKUP was a lifesaver), Access ("So that is why people using relational databases. Awesome!), and back in Excel ("let's not make this so complicated"), I still find that these Micorosft Office apps are the best tools to get the job done.
What's the cost to not use these tools to get the job done?
Contract work to our ERP vendor to add a special feature for this ordering process: are you kidding me? We'd likely pay tens of thousands of dollars for an unflexible monolithic application with horrendous user experience...and we would still end up back in Excel.
Buy third party software designed for this exact process: I've seen an on-site demo of software that does exactly what I want for our procurement process. It starts at $100,000. There are probably other tools that we can get for a few thousand dollars, but at that price point, I've already emulated most of their features in my own application.
Try to finish the job "by hand." : Ha! I'm a programmer at heart, which means I'm lazy. If it takes a solid week of sitting at a desk to work up a purchase order (it actually did take this long), you can bet I'm going to work up a solution so that it only takes me a few hours (and now it does). Perhaps the guy after me will go back to doing most of it by hand, but I'll use the tools in my toolbox to save myself time and stress.
It's so hard to find the perfect application to allow for maximum creativity on the user end but still allow IT to "manage" it. Once you think you've found a solution for one thing, you realize it doesn't do something else. Can I write I printable report in this solution like I used to do in Access? Can I write complicated Excel formulas that tie multiple data sources together from different sheets ("You want me to learn what? No, I've never heard of a "SQuirreL query" before. VLOOKUP is just fine thankyouvermuch)? Can I e-mail the results to the people in my department? Can it automatically pull data from our back-end database like I do in Excel and Access? Can I write my own code, VBA or otherwise, to make my job easier? The list goes on.
In the end, the best advice I can give to any IT manager in your situation is to respect the other workers at your company. Let them know their work is important (even if it's only useful to them and the guy at the next desk over). Let them know you are not trying to make their job harder. Don't assume they are morons for creating mission-critical applications in office productivity software; they are just trying to get the job done with the tools at hand and are usually quite capable and intelligent people. Invite them to explore different solutions with you instead of just removing the tools they currently have in their toolbox and then replacing them with ones they don't know how to use.
At the end of the day, if you have users who are smart enough to shoot themselves in the foot by creating complicated apps in Excel and Access, they are probably smart enough to learn to use the appropriate tools to accomplish the same tasks. Invest the time and energy to involve them in the process and you will have a solution that works for everyone at the end.
You could try a hybrid approach: Allow your users to use Excel/Access to home-brew their own, specialized tools, but take the mission-critical stuff and put it under IT control. There are a few strategies that could help you with this:
Make sure that your IT department is firm on VBA. Not the "yeah-everybody-can-write-a-few-lines-of-basic" type of knowledge, but in-depth training, just like you would if it were a less simple programming language. Although "real programmers" will tell you otherwise, it is possible to write large, stable applications in VBA.
If you currently have the data in Access databases, move away from that and migrate it to an SQL Server. This allows you to do centralized backup and management, while still giving your power users the flexibility to "link" these SQL Server tables to their Access frontend.
Commonly used business logic should be under control of your IT department. This can be done either with VBA, by creating an Access library that is linked by your users, or in any of the .net languages, using COM interop. The latter sounds more complicated than it is, and it will increase the satisfaction of your IT department, since developing in .net is just much more rewarding than VBA (version control possible, etc.).
I would second one of Kevin Ross's main points:
I personally think their should be a
small team of people within IT whose
job (or one of their jobs) is to
develop these small applications. They
should work very closely with the end
users and not be locked in the ivory
tower of IT.
I think any IT department that has a lot of users using Access/Excel should have at least one properly trained and experienced specialist in developing apps on those platforms. That person would be the go-between to make sure that:
IT's priorities and policies get properly implemented in the home-grown apps.
the end users get expert help in converting their home-grown efforts into something more stable and well-designed.
I would second Tony's point that whoever works with the end users in revising these apps to meet IT standards should work side-by-side with the users. The Access/Excel specialist should be an advocate for the end users, but also for the IT policies that have to be followed.
I also think that an IT department could have a specialist or two on staff, but should also have a full-time professional Access and/or Excel developer as a consultant, since the on-staff people could probably handle day-to-day issues and management of the apps, while the professional consultant could be called in for planning and architecture and for the implementation of more complex feature sets.
But all of that would depend on the size of the organization and the number of apps involved. I don't know that it would be desirable to have someone on salary who is nothing but an Access/Excel specialist, precisely because of the problem you get with all salaried employees compared to consultants -- the employees don't see as wide a variety of situations as an active consultant with the same specialization is likely to see and thus the consultant is going to have broader experience.
Of course, I recognize that many companies do not like to outsource anything, or not something that important. I think that's unwise, but then again, I'm the person that gets hired by the people who decide to do it!
If it's mission critical, and it's in Access or Excel, is built poorly, and no one understands it, it is probably time to rebuild it properly.
When the 'users' are in control it usual means one particular person is in control of the architecture, design, coding and documentation... except they normally omit the documentation step. Source control and bug reporting, the touchstone of software development, is usually absent. Few instances of code reuse, due to the nature of Office apps (code modules usually embedded into documents) and VBA (little OOP, most VBA coders don't use Implements, etc). All this means that the resulting applications are not subject to get proper scrutiny and quality can suffer, meaning there are likely to be maintenace issues, escpecially when that one user leaves. I know because I used to be that person ;)
So in order to satisfy the IT department, the proper process needs to be applied. That one 'power' user can continue to own the design and coding but will get peer review, perhaps the serivces of a technical author and a dedicated tester, be required to use source control, perhaps consider integrating with enterprise systems, etc.
There is no getting around the use of Excel/Access. It's what's available, and still very powerful and flexible. The best thing to do is offer some guidelines as to how files should look and be set up. If everyone is using similar standards then the files will live longer and more productive lives, beyond the creator's tenure at the company.
You've got some excellent answers regarding dealing with the folks and the business side of things. So my response will be more technical.
If you are going to redesign the app have the developers work in the same offices as the users. Given the users updates every day or two. If the users have any minor suggestions give those to the users within a day or two. Ultra Frequent Application Deployment
Give the power users an Access MDB/ACCDB linked to the tables with a bunch of starter queries. Let them create the queries they need to export the data to Excel for their own purposes and distribution to clients.
Today online security is a very important factor. Many businesses are completely based online, and there is tons of sensitive data available to check out only by using your web browser.
Seeking knowledge to secure my own applications I've found that I'm often testing others applications for exploits and security holes, maybe just for curiosity. As my knowledge on this field has expanded by testing on own applications, reading zero day exploits and by reading the book The Web Application Hacker's Handbook: Discovering and Exploiting Security Flaws, I've come to realize that a majority of online web applications are really exposed to a lot of security holes.
So what do you do? I'm in no interest of destroying or ruining anything, but my biggest "break through" on hacking I decided to alert the administrators of the page. My inquiry was promptly ignored, and the security hole has yet not been fixed. Why wouldn't they wanna fix it? How long will it be before someone with bad intentions break inn and choose to destroy everything?
I wonder why there's not more focus on this these days, and I would think there would be plenty of business opportunities in actually offering to test web applications for security flaws. Is it just me who have a too big curiosity or is there anyone else out there who experience the same? It is punishable by law in Norway to actually try break into a web page, even if you just check the source code and find the "hidden password" there, use it for login, you're already breaking the law.
I once reported a serious authentication vulnerability in a online audiobook store that allowed you to switch the account once you were logged in. I was wary too if I should report this. Because in Germany hacking is forbidden by law too. So I reported the vulnerability anonymously.
The answer was that although they couldn’t check this vulnerability by themselves as the software was maintained by the parent company they were glad for my report.
Later I got a reply in that they confirmed the dangerousness of the vulnerability and that it was fixed now. And they wanted to thank me again for this security report and offered me an iPod and audiobook credits as a gift.
So I’m convinced that reporting a vulnerability is the right way.
"Ive found that Im often testing others applications for exploits and security holes, maybe just for curiosity".
In the UK, we have the "Computer Misuse Act". Now if these applications you're proverbially "looking at" are say Internet based and the ISP's concerned can be bothered to investigate (for purely political motivations) then you're opening yourself up getting fingered. Even doing the slightest "testing" unlesss you are the BBC is sufficient to get you convicted here.
Even Penetration Test houses require Sign Off from companies who wish to undertake formal work to provide security assurance on their systems.
To set expectations on the difficulty in reporting vulnerabilties, I have had this with actual employers where some pretty serious stuff has been raised and people have sat on it for months from the likes of brand damage to even completely shutting down operations to support an annual £100m E-Com environment.
I usually contact the site administrator, although the response is almost ALWAYS "omg you broke my javascript page validation I'll sue you."
People just don't like to hear that their stuff is broken.
Informing the administrator is the best thing to do, but some companies just won't take unsolicited advice. They don't trust or don't believe the source.
Some people would advise you to exploit the security flaw in a damaging way to draw their attention to the danger, but I would recommend against this, and it's possible that you could have serious consequences because of this.
Basically if you've informed them it's no longer your problem (not that it ever was in the first place).
Another way to ensure you get their attention is to provide specific steps as to how it can be exploited. That way it will be easier for whomever recieves the email to verify it, and pass it on to the right people.
But at the end of the line, you owe them nothing, so anything you choose to do is sticking your neck out.
Also, you could even create a new email address for yourself to use to alert the websites, because as you mentioned, some places it would be illegal to even verify the exploit, and some companies would choose to go after you instead of the security flaw.
If it doesn't affect many users, then I think notifying the site administrators is the most you can be expected to do. If the exploit has widespread ramifications (like a Windows security exploit) then you should notify someone in a position to fix the problem, then give them time to fix it before you publish the exploit (if publishing it is your intention).
A lot of people cry about exploit publication, but sometimes that's the only way to get a response. Keep in mind that if you found an exploit, there's a high likelihood that someone with less altruistic intentions has found it and has started exploiting it already.
Edit: Consult a lawyer before you publish anything that could damage a company's reputation.
I experienced the same like you. I once found an exploit in an oscommerce shop where you could download ebooks without paying. I wrote two mails:
1) Developers of oscommerce, they answered "Known issue, just don't use this paypal module, we won't fix"
2) Shop administrator: no answer at all
Actually I have no idea what's the best way to behave ... maybe even publicate the exploit to force the admins to react.
Contact the administrator, not a business-type person. Generally the admin will be thankful for the notice, and the chance to fix the problem before something happens and he gets blamed for it. A higher-up, or the channels a customer service person is going to go through, are the channels where lawyers get involved.
I was part of a group of people who reported an issue we stumbled across on the NAS system at University. The admins were very grateful we found the hole and reported it, and argued with their bosses on our behalf (the people in charge wanted to crucify us).
We informed the main developer about a sql injection vulnerability on their login page. Seriously, it's the classic '<your-sql-here>-- variety. You can't bypass the login, but you can easily execute arbitrary sql. Still hasn't been fixed in 2 months! Not sure what to do now...no one else at my office really cares, which amazes me since we pay so much for every little upgrade and new feature. It also scares me when I think about the code quality and how much stock we are putting in this software.