We are evaluating various error reporting services for iOS and Android apps. Our app uses webservices to connect to the server.
We currently use Flurry analytics but have not yet used their new error reporting feature.
What is your feedback on Flurry error reporting, if you are using that today?
I am trying to compare it with Crittercism and Crashlytics. So, if anyone out there have experience using Flurry error reporting and Crittercism or Crashlytics, it would be great to hear your feedback.
Thanks.
While this question is likely more suited for a forum, I'll do my best to answer your question from my experience with these technologies as well as what I've heard from others. First lets look at what these services provide for Crash Reporting, then lets look at what they provide that ties into Crash Reporting.
Flurry provides (at a high level):
Analytics of event based metrics (this includes optional basic crash reporting)
server-side symbolication (however uploading symbols is a manual process)
All this is great, but they really don't focus on crash reporting at all or how that ties into other metrics.
Crittercism provides (at a high level):
real-time insights on reports (system diagnostics, logs, etc)
smart crash/error grouping
error monitoring with actionable diagnostic information
logging non-volatile errors
server-side symbolication for stack traces
location correlation with above metrics
session based metrics for a crash report
client perspective network performance data (note: this would be great to keep track of your web services and other 3rd party services, such as Flurry, to see how they're performing)
trends of above metrics and data
connection between network calls and crash/error reports (look at breadcrumbs)
See the above link for more details on the full package that Crittercism provides. For more details on some of the features, check out this page.
All of these are focused on giving you full visibility on how your app is performing for each of your users.
Crashlytics provides (at a high level):
smart crash/error grouping
error monitoring with actionable diagnostic information
server-side symbolication for stack traces
logging non-volatile errors
real-time insights on reports (system diagnostics, etc)
Flurry can be paired with more robust solutions for Crash Reporting and in my experience most end up taking this route.
Crittercism provides a lot more than just Crash Reporting that ties back to optimizing the performance of your applications with actionable data. They also hook into Mobile-ready support systems (such as UserVoice and HelpShift) for better customer communication, and several task management systems for engineering (JIRA, Github, Pivotal). Server-side symbolication, APIs to pull data, simple integration with build tools (such as Jenkins) provide a much more mature solution.
Crashlytics provides an easy integration for the average developer. I've heard complaints from developers with more defined build processes (such as something that uses Jenkins) that their app integration for uploading symbols for symbolication gets troublesome.
They also provide integrations with JIRA and Github. Not sure at what level.
Some love the UI for Crashlytics, whereas others have stated that it gets in the way.
Hope this helps.
I have been using Crashlytics pretty much since it first launched for Android and iOS. It's a pretty impressive system with some massive pros:
Extremely lightweight library
Compatible with major development environments
Real time insights
Crash grouping
Individual crash breakdown
Tagged with user information
Custom keys/logging
Email alerts
Impact level changes
New bugs
Daily summary reports
User management for sharing access
It's free
There is also a split between crashes and non-fatals which means you can report handled exceptions back without having the app crash. For example, in our Logging class we have this method which gets called when an exception could happen, but shouldn't. This means that we have a breakdown of how often this happens and we can work towards resolving it:
// Android
public static void log_wtf(Throwable throwable)
{
Log.wtf("Log", throwable.toString());
// Also log this exception to Crashlytics
Crashlytics.logException(throwable);
}
I particularly like the crash breakdown which lets you see each individual crash report from a group of reports. You can ask the users to use their name, or do what we do and sign it with their login credentials for our service (see the user info on the right):
This is invaluable information for us which allows us to easily and quickly reproduce the error.
You also get a version breakdown which you can disable at a later point. For example, if we have just finished a test of alpha version 0.1 (1), you can turn logging off for that version and launch version 0.2 (2). If someone forgets to update their application you will not be notified of anything from that point. Obiviously this isn't good when your app is in production, but useful for testing phases.
While I haven't tested it they have integrations for other services including your own web hooks if you wish to develop something to support it. It's in my plans to do this to automatically create trac bug tickets, however I haven't got around to it yet!
Oh, and finally, let's not forget the sweet animations crashlytics employs throughout it's software.
I would recommend going with Crashlytics. They have the smoothest onboarding process of any error reporting software. It's very easy to setup and incredibly intuitive to use. The reporting is immediate and spot on. They also have the best console UI of the bunch. I've used them now at Evernote, HomeAway, and my own apps and I've never ran into any problems. They also have a great support team that responds very very quickly, normally within minutes to inquiries.
I've also used Flurry in the past, for event tracking, with mixed results. Their numbers always seemed to be a bit off. You also have to go through and add start/stop events in every activity which can be a pain.
I exclusively use Crashlytics for my day to day app crash reporting. The UI at times can be a little flashy and navigating the site is sometimes a hassle but overall they provide a reliable and well built product. It's certainly useful when distributing app betas and as an intermediate to advanced iOS developer, I've never had any problems integrating their SDK.
Pros:
immediate crash reports
real time insights
interaction time is minimal i.e. the time spent once a crash email is received to the bug being fixed is small. On average I spend 30 seconds - 5 minutes on every crash report simply because Crashlytics makes it very easy to find my bugs.
great customer support
installation is a breeze
lightweight
Cons:
UX on the website isn't as refined as it could be
navigating the website takes some getting used to.
I work for Grid, an iOS app startup based in San Fran and we use Crittercism there but all in all, Crashlytics is the one I end up using. It comes down to ease of use and they've got that nailed down.
EDIT:
Crashlytics was also recently bought by Twitter so there's plenty of talent and infrastructure there.
I've not used Crittercism or Flurry for crash/error reporting, but have been using Crashlytics since its inception. I'm not convinced there's a better crash reporting solution at this point.
Reasons why I think Crashlytics is great:
Super lightweight
Super easy integration
Awesome stack unwinding
Ability to provide custom logging
Web UI response has greatly improved
Provides symoblicated stack trace
Smart culling of crashes on web UI
Easy crash searching on web UI
Basic stats via web UI
At the end of the day, it comes down to what tool works best for you.
As said before Flurry don't focus on crash reporting at all or how that ties into other metrics
Before Crashlytics android's release I was using Flurry but had some problems with documentation and support (you can see here) and they didn't corresponded as I wished.
In my case, crash and errors are a very critical point cause, so Crashlytics worked like a charm.
A good documentation
Great support
Easy Configuration
Real time insights (Flurry hasn't)
In short, I have not experienced nothing better than Crashlytics yet (Talking about Error Reporting)
I use Crashlytics since a year now (edited July 2014), and the system is really awesome.
Realtime crash report works very well, with their reports we are able to fix them really quickly, it's a huge & priceless feature ! CL solution is available on multiple platforms, it's really easy to plug it in XCode 5, Eclipse or IJIdea with less than 5 lines of code.
You can create multiple enterprise, to separate your projects, just with a couple of clicks
I have to mention that support team is fabulous, I've exchange dozen of mail about it and their answer always solve my little probs.
It's completely free, own now by Twitter, and very very light.
I gave a conference about Craslytics solution a month ago, slides are available on my speakerDeck profile here.
Related
I want to build a constantly running forex trading application (even if the web page is not opened),
But I don't know what is the best way to do it, it's a software that should run constantly with constant internet connection, should I write it in Python as a PC application or web application ?, or is there anything better ?
Thank you for your support,
I don't see any point of making a web application in your case. Architecture of algorithmic trading systems is a broad subject but most of the time it's an application which is only connected to a market data provider and a broker. A trading system can be built as a web application for example for browsing portfolio, historical trades or when some external events trigger trades but you should focus on the strategy itself rather than on a time consuming UI.
You should learn some basics from books/online course depending on your programming language but I think that Python is a great choice for rapid prototyping of strategies. You can also try some existing platforms like Quantopian which can give a quick start for strategies testing.
Focus your efforts on thorough testing of the strategy because most of the time they look good on a paper but turn out to be an utter garbage in practice and without a strategy there is no need for an infrastructure.
There are lots of technologies for developing "always on" scanners; I won't scour the internet for you; however, to give you an example, you can use something like Azure Functions -- it can run your scanner on a schedule. It can be built with Python, if that's your language of choice.
Of course, there are a lot of third-party tools to choose from as well.
We have been looking at automatically logging all unexpected client errors to our bug tracker. For reference our application is written in Java/GWT/Guice/Hibernate/Jetty and our bug tracker is the hosted version of FogBugz which can create bugs programatically or via an email.
The biggest problem I see with doing this is stack traces that happen in a loop overload the bug tracker by creating thousands of cases. Does anybody have a suggested way to handle automatic bug creation like this?
If you're using FogBugz bugscout (also see up-to-date docs here) then it has the ability to just increase number of occurences of same problem, instead of creating new case for same exception again and again.
Are you sure that you want to do that?
It obviously depends on your application but even by carefully taking care of the cases that could generate lots of bug reports (because of the loops) this approach could still end up filling the bug tracker.
How about this?
Code your app so that every time an exception is thrown, you gather info about the client (IP, login, app version, etc) and send that + the stack trace (or the whole exception object .ToString()) by email to yourself (or the dev team).
Then on you email client, have a filter that sorts that incoming mail and throws it in a nice folder for you to look at later.
Thus you can have tons of emails about maybe one of more issues but then you don't really care because you input the issues yourself in the bugtracker, and easily delete that ton of mail.
That's what I did for my app (which is a client-server desktop app). It plays out well in this case.
Hope that helped!
JIRA supports automated issues creation using so called services: documentation.
Does anybody have a suggested way to handle automatic bug creation...?
Well, I have. Don't do that.
What are you going to gain from that? Tester's effort? in my experience, whatever effort one can save from that was lost multiple times with overhead transferred to developers who had to analyze and maintain the automatically created tickets anyway. Not to mention overall frustration caused by that.
The least counterproductive way I can imagine would be something like establishing a dedicated bugs category or issue tracker instance, such that only testers can see and use it.
In that "sandbox", auto-created bugs could be assigned to testers who would later pass analyzed and aggregated bug reports to developers.
And even in that case, I'd recommend to pay close attention to what users (testers) say about the system. If they, say, start complaining about the system, consider trying a manual way of doing things instead.
How can i test my local RIA?
I need to do a stress test, graph response time and memory usage when user increases.
Do you know any software?
RIA tool support is often dictated by the development platform. For instance if you have GWT and need Javascript support in the tool then you will be pushed to one subset of tools, Silverlight to another, etc...
Looks to your development team, System Requirements Document and Architecture documentation for information on the developmnent toolkits used by your rick internet application. Once you have good insight there, into both which toolkit and what version then take a look at the commercial and open source tools out there to see which ones support your interface. There are few things more frustrating than driving a nail with the butt end of a screwdriver, but if your tool and your interface are a poor match you could wind up doing just that.
All of the commercial vendors are offering short term licenses at this point that you should be able to tie directly back to the project budget. Something to keep in mind on the open source front is that the level of effort on the labor front tends to be higher overall because of the efficiencies built into the commercial tools on the development, monitoring integration and analysis fronts.
If you want an open source solution, I can think on Apache JMeter. There are others like Rational Performance Tester or Mercury LoadRunner but those are not free. You might want to verify if there's a trial version out there.
We want to regularly test our web apps for vulnerabilities - we have a BurpSuite licence but would like to look into the possibility of using an online solution.
Benefits as we see them:
Little/no staff training required
It's always up to date (checking the latest known vuns)
It may be more cost-effective
Has anyone used such a solution before? How was the experience over manual testing with a local app such as BurpSuite?
Does your regular testing involve running purely automated scans or manual analysis using BurpSuite?
A cloud-based scanner isn't going to offer as much interactivity as running BurpSuite locally. Manual analysis is important for analyzing design issues, authorization problems, or logic errors. What a fully automated scanner will (or should) do is take care of tedious tests for vulns like SQL injection and XSS. (Think of it as finding implementation problems rather than design problems.)
A cloud-based scanner could be a drop-in replacement for BurpSuite scans you're running "out of the box" -- i.e. not using much configuration. But it should never the less complement, not replace, good manual testing. This can be a benefit cost-wise.
While a cloud-based scanner can be "always up to date", in reality there are very few new web vulns. For example, the OWASP Top 10 has only changed a little in the last 7 years. The updated vulns will help if you're mostly using open source or commercial web apps, but they won't have any bearing on the apps you build in house.
(FYI, my perspective comes from working on a cloud-based scanner.)
My company is a custom development shop for a number of projects, some larger and some smaller. Currently we handle all of our client communication through email. So we email a design doc, they mark it up and send it back. Then we roll out a beta version of their product and they email us with any bugs, new features, etc. And so on....
As I am working on implementing a new bug tracking system (it looks like it will be Mantis right now), I got to wondering how we could best allow our customers an interface with our development process that would provide better tracking of feature requests and client submitted bugs as well as communicate our responses back to the client.
If anyone is aware of a a bug tracking system that does this exceptionally well I'd be interested to hear of that. Otherwise I'm just looking for some general guidelines or good business practices that have allowed your companies to interface effectively and efficiently with your clients.
UPDATE: My company uses a LAMPP stack and as we are a small shop with a limited budget we tend to stick to tools that are open-source and free.
Do most people either use Team Foundation Server to handle this or emails back and forth?
I think the key is to have the dedicated tracking system there for bugs/requests, and to establish a set process for communication. With that at minimum you will start getting consistent feedback. From there you can tweak it to get your specific needs.
As an aside, rather than just using e-mail for your communication, I strongly recommend going to smething like BaseCamp for a project management tool. I find that it helps greatly with keeping messages, documentation, and timelines communicated to the client.
If you are using Team Foundation Server, I recommend you to install TeamPlain Web Access. They allow you to expose a web interface to your TFS project. The only things left to do, is give rights to your client and a username and a password.
Otherwise, there is some paying tools like FogBugz. Of course, the principal is having to bug reporting tools directly linked to your Source Control so that the developers can easily fix bugs.
Although I know of no specific tools (at least no open source ones), I suggest that you setup a system which will cover your overall requirements gathering and implementation process. Requirements could be tracked in the system, which would also contain the design documents (which could be "checked out from" and "committed to" the system). This way, you would tackle the problem of having multiple revisions of design documents around. Addionally, the design documents and the requirements could be tracked easily. If this system were linked to your source code management system, you would additionally ease your development process/requirements tracking.
Another possibility is to use two products in concert, here's our current setup with a team of 12:
osTicket for incoming requests from clients
Allows for issues to be handled by support staff and bugs to be verified
Status can be checked with just an email address and ticket ID
Typically users don't submit detailed enough bug reports so is a good first step
redmine for development tickets
Ticket created by QA or a developer if issue is a real bug
Provides solid enough project and release management
Is a solid step up from trac and mantis (and provides migration tools)