We have been looking at automatically logging all unexpected client errors to our bug tracker. For reference our application is written in Java/GWT/Guice/Hibernate/Jetty and our bug tracker is the hosted version of FogBugz which can create bugs programatically or via an email.
The biggest problem I see with doing this is stack traces that happen in a loop overload the bug tracker by creating thousands of cases. Does anybody have a suggested way to handle automatic bug creation like this?
If you're using FogBugz bugscout (also see up-to-date docs here) then it has the ability to just increase number of occurences of same problem, instead of creating new case for same exception again and again.
Are you sure that you want to do that?
It obviously depends on your application but even by carefully taking care of the cases that could generate lots of bug reports (because of the loops) this approach could still end up filling the bug tracker.
How about this?
Code your app so that every time an exception is thrown, you gather info about the client (IP, login, app version, etc) and send that + the stack trace (or the whole exception object .ToString()) by email to yourself (or the dev team).
Then on you email client, have a filter that sorts that incoming mail and throws it in a nice folder for you to look at later.
Thus you can have tons of emails about maybe one of more issues but then you don't really care because you input the issues yourself in the bugtracker, and easily delete that ton of mail.
That's what I did for my app (which is a client-server desktop app). It plays out well in this case.
Hope that helped!
JIRA supports automated issues creation using so called services: documentation.
Does anybody have a suggested way to handle automatic bug creation...?
Well, I have. Don't do that.
What are you going to gain from that? Tester's effort? in my experience, whatever effort one can save from that was lost multiple times with overhead transferred to developers who had to analyze and maintain the automatically created tickets anyway. Not to mention overall frustration caused by that.
The least counterproductive way I can imagine would be something like establishing a dedicated bugs category or issue tracker instance, such that only testers can see and use it.
In that "sandbox", auto-created bugs could be assigned to testers who would later pass analyzed and aggregated bug reports to developers.
And even in that case, I'd recommend to pay close attention to what users (testers) say about the system. If they, say, start complaining about the system, consider trying a manual way of doing things instead.
Related
We are evaluating various error reporting services for iOS and Android apps. Our app uses webservices to connect to the server.
We currently use Flurry analytics but have not yet used their new error reporting feature.
What is your feedback on Flurry error reporting, if you are using that today?
I am trying to compare it with Crittercism and Crashlytics. So, if anyone out there have experience using Flurry error reporting and Crittercism or Crashlytics, it would be great to hear your feedback.
Thanks.
While this question is likely more suited for a forum, I'll do my best to answer your question from my experience with these technologies as well as what I've heard from others. First lets look at what these services provide for Crash Reporting, then lets look at what they provide that ties into Crash Reporting.
Flurry provides (at a high level):
Analytics of event based metrics (this includes optional basic crash reporting)
server-side symbolication (however uploading symbols is a manual process)
All this is great, but they really don't focus on crash reporting at all or how that ties into other metrics.
Crittercism provides (at a high level):
real-time insights on reports (system diagnostics, logs, etc)
smart crash/error grouping
error monitoring with actionable diagnostic information
logging non-volatile errors
server-side symbolication for stack traces
location correlation with above metrics
session based metrics for a crash report
client perspective network performance data (note: this would be great to keep track of your web services and other 3rd party services, such as Flurry, to see how they're performing)
trends of above metrics and data
connection between network calls and crash/error reports (look at breadcrumbs)
See the above link for more details on the full package that Crittercism provides. For more details on some of the features, check out this page.
All of these are focused on giving you full visibility on how your app is performing for each of your users.
Crashlytics provides (at a high level):
smart crash/error grouping
error monitoring with actionable diagnostic information
server-side symbolication for stack traces
logging non-volatile errors
real-time insights on reports (system diagnostics, etc)
Flurry can be paired with more robust solutions for Crash Reporting and in my experience most end up taking this route.
Crittercism provides a lot more than just Crash Reporting that ties back to optimizing the performance of your applications with actionable data. They also hook into Mobile-ready support systems (such as UserVoice and HelpShift) for better customer communication, and several task management systems for engineering (JIRA, Github, Pivotal). Server-side symbolication, APIs to pull data, simple integration with build tools (such as Jenkins) provide a much more mature solution.
Crashlytics provides an easy integration for the average developer. I've heard complaints from developers with more defined build processes (such as something that uses Jenkins) that their app integration for uploading symbols for symbolication gets troublesome.
They also provide integrations with JIRA and Github. Not sure at what level.
Some love the UI for Crashlytics, whereas others have stated that it gets in the way.
Hope this helps.
I have been using Crashlytics pretty much since it first launched for Android and iOS. It's a pretty impressive system with some massive pros:
Extremely lightweight library
Compatible with major development environments
Real time insights
Crash grouping
Individual crash breakdown
Tagged with user information
Custom keys/logging
Email alerts
Impact level changes
New bugs
Daily summary reports
User management for sharing access
It's free
There is also a split between crashes and non-fatals which means you can report handled exceptions back without having the app crash. For example, in our Logging class we have this method which gets called when an exception could happen, but shouldn't. This means that we have a breakdown of how often this happens and we can work towards resolving it:
// Android
public static void log_wtf(Throwable throwable)
{
Log.wtf("Log", throwable.toString());
// Also log this exception to Crashlytics
Crashlytics.logException(throwable);
}
I particularly like the crash breakdown which lets you see each individual crash report from a group of reports. You can ask the users to use their name, or do what we do and sign it with their login credentials for our service (see the user info on the right):
This is invaluable information for us which allows us to easily and quickly reproduce the error.
You also get a version breakdown which you can disable at a later point. For example, if we have just finished a test of alpha version 0.1 (1), you can turn logging off for that version and launch version 0.2 (2). If someone forgets to update their application you will not be notified of anything from that point. Obiviously this isn't good when your app is in production, but useful for testing phases.
While I haven't tested it they have integrations for other services including your own web hooks if you wish to develop something to support it. It's in my plans to do this to automatically create trac bug tickets, however I haven't got around to it yet!
Oh, and finally, let's not forget the sweet animations crashlytics employs throughout it's software.
I would recommend going with Crashlytics. They have the smoothest onboarding process of any error reporting software. It's very easy to setup and incredibly intuitive to use. The reporting is immediate and spot on. They also have the best console UI of the bunch. I've used them now at Evernote, HomeAway, and my own apps and I've never ran into any problems. They also have a great support team that responds very very quickly, normally within minutes to inquiries.
I've also used Flurry in the past, for event tracking, with mixed results. Their numbers always seemed to be a bit off. You also have to go through and add start/stop events in every activity which can be a pain.
I exclusively use Crashlytics for my day to day app crash reporting. The UI at times can be a little flashy and navigating the site is sometimes a hassle but overall they provide a reliable and well built product. It's certainly useful when distributing app betas and as an intermediate to advanced iOS developer, I've never had any problems integrating their SDK.
Pros:
immediate crash reports
real time insights
interaction time is minimal i.e. the time spent once a crash email is received to the bug being fixed is small. On average I spend 30 seconds - 5 minutes on every crash report simply because Crashlytics makes it very easy to find my bugs.
great customer support
installation is a breeze
lightweight
Cons:
UX on the website isn't as refined as it could be
navigating the website takes some getting used to.
I work for Grid, an iOS app startup based in San Fran and we use Crittercism there but all in all, Crashlytics is the one I end up using. It comes down to ease of use and they've got that nailed down.
EDIT:
Crashlytics was also recently bought by Twitter so there's plenty of talent and infrastructure there.
I've not used Crittercism or Flurry for crash/error reporting, but have been using Crashlytics since its inception. I'm not convinced there's a better crash reporting solution at this point.
Reasons why I think Crashlytics is great:
Super lightweight
Super easy integration
Awesome stack unwinding
Ability to provide custom logging
Web UI response has greatly improved
Provides symoblicated stack trace
Smart culling of crashes on web UI
Easy crash searching on web UI
Basic stats via web UI
At the end of the day, it comes down to what tool works best for you.
As said before Flurry don't focus on crash reporting at all or how that ties into other metrics
Before Crashlytics android's release I was using Flurry but had some problems with documentation and support (you can see here) and they didn't corresponded as I wished.
In my case, crash and errors are a very critical point cause, so Crashlytics worked like a charm.
A good documentation
Great support
Easy Configuration
Real time insights (Flurry hasn't)
In short, I have not experienced nothing better than Crashlytics yet (Talking about Error Reporting)
I use Crashlytics since a year now (edited July 2014), and the system is really awesome.
Realtime crash report works very well, with their reports we are able to fix them really quickly, it's a huge & priceless feature ! CL solution is available on multiple platforms, it's really easy to plug it in XCode 5, Eclipse or IJIdea with less than 5 lines of code.
You can create multiple enterprise, to separate your projects, just with a couple of clicks
I have to mention that support team is fabulous, I've exchange dozen of mail about it and their answer always solve my little probs.
It's completely free, own now by Twitter, and very very light.
I gave a conference about Craslytics solution a month ago, slides are available on my speakerDeck profile here.
(as at 1st June 2012)
A very simple question I would really appreciate input from others (I think we all could) who are either wondering the same or have decided to go with it (if so, any user experiences shared would be great).
I have 2 main issues which seem common on stackoverflow and on various blogs, so I know I'm not the only person experiencing them.
Occasionally my sync gets corrupted for one reason or another and the sync either fails on certain data or fails to sync altogether.
Pre populating data is essential for my app, there's no reliable way of doing this (without asking the user - which isn't reliable)
I've seen many good work arounds to both these issues but I can't help feeling that my users will encounter problems still.
I'd be really interested to learn of any experiences particularly from those that have decided to publish using iCloud and Core-Data.
iCloud has been around a while now but even on iOS 5.1.1 it doesn't seem stable enough, surely a more reliable version can't be too far away.
It is hard to answer this one without "Making statements based on opinion", but it's one worth giving partially backed up opinions on to hopefully save others some time and frustration.
My opinion is that it's just not ready. I unfortunately have an app in the wild at the moment based around CoreData + iCloud and even though we danced around a number of iCloud bugs to finally get it stable enough to pass reasonably thorough testing some users are still running into the corrupted sync states. I've spent a healthy portion of my recent life trying to make it work and am currently reluctantly reimplementing the app without Core Data. I really hope that it stabilizes soon as I genuinely like most of it.
At the moment support requests / bug reports made by customers are coming in by mail. It is getting harder to organize priorities and stay at the helm of all this. So I am looking for bugtracking(?) tools. Not all reports are bugs of course, sometimes it's just feature request or support requests.
So my question is: whicht open source bugtracker / support request handling tool do you recommend? I know Mantis which seems to be my front runner for a more elaborate evaluation, but I already worked with it (as a reporter / contributor) and found the GUI a little cumbersome. Another issue is that I thought about using the tool for multiple website projects of different customers.
Intuitively I would prefer to run only one instance of the tool for all projects to have a better overview of all critical issues (independently of the project). Of course customer A should not be able to see customer Bs request (but every customer can have multiple reporting accounts) Is Mantis able to handle that? Can you recommend any other alternatives?
P.S.: I heard about Jira, but I will try to find a free tool for my first try.
It's possible to use email with Mantis, so that you can get incoming email (directly or by forwarding) to Mantis.
Then you can have a workflow in Mantis, f.ex. have an incoming project and customer projects, and you can send email with bcc Mantis and subject containing issue number (I use [1234] as a pattern).
I haven't used other issue trackers as much, but my experience with a customized Mantis is good regarding different kinds of issues and using with email.
Since you're turned to Open source, I'd say install a project management platform like Launchpad, redmine... etc and then create a project for each of your clients (of course you can have multiple accounts for only one client). The bug tracker in these platforms can serve as a support request service.
I'd go for Launchpad because it also has the Q/A feature and blueprints, and is also nice looking and very very user-friendly. And also damn easy to install on a Ubuntu Server.
Kind regards
Did you ever find yourself in this position - a bug is reported by the client but you are unable to reproduce the bug. Even the QA team is unable to reproduce the bug. We are facing such a situation with our desktop application(C#, windows forms). What do you suggest we should do to reproduce the bug, or track the scenario which generates the bug.
Yes, this is a common situation.
I find that the best way to handle this situation is to log as much as possible in log files and get the customer to send the log files to you for analysis, as well as their description of what they were doing when the problem occurred. The customer description is often very incomplete and misses out important details about what they were doing when the error happened. A detailed log can help fill out the holes in the story.
If you are lucky enough that the log file includes a stack trace it is often possible to reason about possible causes of the error even if it is a rarely occuring bug that you have not been able to reproduce yourself. In this situation a code review of the affected code can reveal flaws in the design of the code and suggest an alternative approach that is less likely to fail.
This all dependes on what kind of bug you are facing (Logic, User Interface, Multi Threaded, Or actual User).
Ensure that the client has given you
all the required info to reproduce
the bug.
If at all possible, have the client
show you how to reproduce the bug, or at the least send some screen shots of how it was done.
Ensure that you write some Unit Test
to try and cover the bug, try proving that it is there.
For some time I have been looking at the possibility to integrate PowerShell as a scripting engine in SharePoint but I haven't found the right solution yet.
My main objective is to enable event triggers in e.g. a list to call and execute a PowerShell script (by filename) on the local server. This would give me a lot of flexibility compared to using an ordinary event handler written in visual studio, but the question is whether it is possible and whether I have overlooked some serious security issues?
Since each and every unique idea that I come up with in many years have already be invented by somebody else, I might have missed an existing product/project so any links to such projects will be appreciated, thanks
In the spirit of "already being invented by somebody else", check out http://www.codeplex.com/iLoveSharePoint for some very interesting uses of PowerShell inside SharePoint. Some great code samples and documentation. Haven't tried myself yet, but seems interesting.
I see what you're trying to achieve, but there's something that just doesn't "feel right" about a user indirectly running script code on your server.
The key difference is that the script can be run by anyone logging into the server. Event handlers can only be run by SharePoint. Strict validation of any inputs would be essential. You should also ensure the script is signed so tampered scripts won't execute.
Also, scripts by their nature aren't really designed for enterprise solutions. There is less opportunity for best practices such as good software architecture, design patterns, source control, code analysis, unit testing, and reuse of code. It's also messy/difficult to share code with a common code base that contains web parts, controls, entities, etc.
Finally, introducing PowerShell means another technology to be maintained in the mix we already have with SharePoint. This might be OK if you are comfortable with it.
Depending on how much customisation has already been done or is planned for the future some of the points above may not matter. Be sure to think about how this idea would feel if implemented 6, 12 and 24 months down the track.