What is a best way to parse an email with bug description. One client decided recently that it would be nice for user to be able to send an email to known mailbox and a bug would be registered in bug tracker (not exactly bu close).
The problem is bug description has lot of fields like dates, times, descriptions, comments, losses, attachments etc. Relying on user to use some specific mail format is not the smartest thing to do.
The question is how could one parse email to get all needed information. The format should be not to strict, but enough to guess what fields mean what. I would also be interested to hear both correct and easiest solutions for this.
P.S.
Actually this feature was requested by a bank. They have a public mailbox where clients would sends discovered issues. The problem is to get as much information from these letters before bank employee will actually look at it.
We do something similar to this with RT, however the email isn't really parsed. All the emails go into a single queue where it is evaluated by our IT staff. Basically, the parsing is done by humans - they modify the ticket to have as much information as they can glean from the email.
You're unlikely to get users to adhere (correctly) to any special syntax or formatting you come up with - unless they are employees, highly trained, or have some incentive to follow your rules.
Another option would be to have the initial email respond with an email that is formatted as a questionaire. In other words, the user writes an initial bug report and immediately (or as soon as your email server can respond) gets back a "thank you - can you provide more info" message with prompts for more info. You could then parse that email and have it populate your bug tracking system with more accurate info,
Good luck! Sounds like a cool idea!
FogBugz has the ability to monitor an email address and add emails sent to that address as a new FogBugz cases.
There's also a feature called ScoutSubmit that accepts HTTP GET arguments and uses those to submit a new case. Very handy for having a application automatically submit bug reports from the field.
Categorizing a bug based on freeform text is a difficult proposition. Very little besides the defect submitter name and the date the bug is reported is easily gleened from an email. Is there a reason you are limiting yourself to email? If you provide a form to submit the bugs via a webpage you can categorize the defect/bug based on dropdown menu items you present to the user. In addition you can point them to common answers in a dynamic information portion of the page. Have a look at Apple iTunes support request page for a slightly annoying but effective method to force the user to give you decent information. Banking applications are not a good domain to allow ambiguity nor are they a good domain to have multiple rounds of communication.
Related
[Problem]
I'm currently having a problem where the same conversation/thread gets allocated to multiple labels/folders. And I want to have a setup where:
If a specific person sends an email or sends an email with a specific subject, it'll go to a specific label.
If someone replies to that conversation, it'll stay in that label that it originally arrived in and it won't appear in any other label. And it wouldn't go to the inbox as well.
[Setup]
I believe my current setup causes the above problem to happen because I have countless filters that uses the same person for From and/or To. So to prevent it as much as possible, I try to use the subject field, to which it works. However, I'm still getting conversations with multiple labels that are not related to any other subjects, so it's very difficult to manage.
[Question]
Is there a way where I could always keep the conversation in the label where it first arrived no matter who or how many people replies to that specific conversation? Should I use apps script? Is there any kind of best practices that you recommend?
I am writing an online survey and I am wondering if there are any good techniques for allowing anonymous people to go to the survey and participate and also prevent duplicates.
I have considered the following, but there are drawbacks from each:
Use cookie in browser
Record IP address
Compare answers for similarities along with either/or the first two methods
Of the techniques I have considered, you either prevent multiple people from using the same device, or make it easy for a user to duplicate survey results. Hopefully someone has an excellent way to prevent this :)
Well I am not sure whether you are thinking of deliberate or accidental duplicates?
if you think people will want to post a load of results to skew the survey, I cannot add anything because any ID-related question you ask can be falsely answered.
if you want people to just give their answers without having to go through a login process, how about asking for their initials plus birthdate (ddmmyyyyfl)- that has a pretty good chance of being unique without really compromising their identities or taking too much time.
Was that what you were after?
Ed
I am currently investigating a similar scenario.
Some of the suggestions I found online are:
You generate a unique URL - which you can send to their email (this email does not have to be stored), and then you add a checksum to the URL to verify it is valid.
Similar to the above mentioned, you provide them with an uniquely generated password, and you validate whether the password has been used before.
The clear limitation is that you require their email, and this is slightly lengthy. However, the email address is not associated with the answer set.
Meaning, you can validate whether an email address has been used to send a URL/Password to. Prevents same email address from being used over and over.
Then when the URL/Password is used, you validate whether that unique reference has been used in an answer set before. (The answer set is associated with the Unique Reference, and not the person's email - ensuring anonymity).
The problem with using email, ID numbers, and Birth Date; is that all of these values can be fabricated.If this approach is used, also do not forget good old CAPTCHA, as a script can be created to run through the combinations and submit answer sets.
I realize this is an old post, but hopefully it helps someone at some point. All of the best.
What if I choose to use GMail's awesome mail archive search capabilities on my database? What if, for every transaction that my database is responsible, I emailed details of that transaction to a GMail address that exists for the sole purpose of searching and retrieving transactions.
Anyone logged into that account could search according to labels, invoice numbers, customer names - whatever using Google's search engine. The results are presented as 'email messages'.
Imagine a user working from the standard (web-based) GMail account searches for an invoice number via GMail's search box - he's returned all instances where the db did anything that included that unique number. Opening any of these 'email messages' would have the static text text included at the time of the transactions (historical and tracking gold) but could also carry a Gadget that could transform the 'message' into an editor so as to execute a new transaction on that invoice.
Imagine further that I wasn't the first one to think of this - cuz surely i'm not - and even if i were, i'm not smart enough to execute the idea alone.
Are you aware of efforts similar to this?
thx
[?belongs on superuser instead?]
An interesting idea, however given your search parameters it might be unreliable. Although gmail's search is great, I have found issues when searching for partial terms. Case in point, I had an email whose subject line was "stuffas". When I searched for "stuffa" I got no results, when I searched for "stuffas" I got the email in the search result. Additionally, I had an email with an 8 digit number inside the body. When I searched for 7 digits out of 8, I got no results, but when I put all 8 digits, the email appeared in the results. So, search in gmail may not be as powerful of a solution as you think. Again this is my experience, I'd love to hear if someone is able to partial search numbers in gmail.
I just had the same idea; 4 years after you. It still doesn't look like this has 'been done before' in any production sense. But now in 2014, I really don't see why not. Python packages for interfacing with gmail are already there and dead-simple to use. It does not take a whole lot of abstraction to turn this into a generalized key-value storage.
Its probably not exactly the fastest database, and not the best solution for everything; but as an easy-to-use, easy to search, trivial to configure, 100% uptime, cloud stored and backed up, free-as-in-beer database, its pretty epic as far as I can see.
Anyone else has seen examples of this having been done before?
Edit: having thought about it some more, there are several answers as to why this is a bad idea:
gmail does not permit random access from different locations; it will block you account. quite a showstopper
amazon simpleDB also gives you a simple key-value store with the same characteristics (plus good python support), and isn't THAT big of a pain to set up if you are willing to spend a day wrapping your head around it. And is also effectively free for the kind of traffic that youd be able to cram into a gmail account.
How can I prevent that forms can be scanned with a sort of massive vulnerability scanners like XSSME, SQLinjectMe (those two are free Firefox add-ons), Accunetix Web Scanner and others?
These "web vulnerability scanners" work catching a copy of a form with all its fields and sending thousands of tests in minutes, introducing all kind of malicious strings in the fields.
Even if you sanitize very well your input, there is a speed response delay in the server, and sometimes if the form sends e-mail, you vill receive thousands of emails in the receiver mailbox. I know that one way to reduce this problem is the use of a CAPTCHA component, but sometimes this kind of component is too much for some types of forms and delays the user response (as an example a login/password form).
Any suggestion?
Thanks in advance and sorry for my English!
Hmm, if this is a major problem you could add a server-side submission-rate limiter. When someone submits a form, store some information in a database about their IP address and what time they submitted the form. Then whenever someone submits the form, check the database to see if it's been "long enough" since the last time that IP address submitted the form. Even a fairly short wait like 10 seconds would seriously slow down this sort of automated probing. This database could be automatically cleared out every day/hour/whatever, you don't need to keep the data around for long.
Of course someone with access to a botnet could avoid this limiter, but if your site is under attack by a large botnet you probably have larger problems than this.
On top the rate-limiting solutions that others have offered, you may also want to implement some logging or auditing on sensitive pages and forms to make sure that your rate limiting actually works. It could be something simple like just logging request counts per IP. Then you can send yourself an hourly or daily digest to keep an eye on things without having to repeatedly check your site.
Theres only so much you can do... "Where theres a will theres a way", anything that you want the user to do can be automated and abused. You need to find a median when developing, and toss in a few things that may make it harder for abuse.
One thing you can do is sign the form with a hash, for example if the form is there for sending a message to another user you can do this:
hash = md5(userid + action + salt)
then when you actually process the response you would do
if (hash == md5(userid + action + salt))
This prevents the abuser from injecting 1000's of user id's and easily spamming your system. Its just another loop for the attacker to jump through.
Id love to hear other peoples techniques. CAPTCHA's should be used on entry points like registration. And the method above should be used on actions to specific things (messaging, voting, ...).
also you could create a flagging system, and anything the user does X times in X amount of time that may look fishy would flag the user, and make them do a CAPTCHA (once they enter it they are no longer flagged).
This question is not exactly like the other questions about captchas but I think reading them if you haven't already would be worthwhile. "Honey Pot Captcha" sounds like it might work for you.
Practical non-image based CAPTCHA approaches?
What can be done to prevent spam in forum-like apps?
Reviewing all the answers I had made one solution customized for my case with a little bit of each one:
I checked again the behavior of the known vulnerability scanners. They load the page one time and with the information gathered they start to submit it changing the content of the fields with malicious scripts in order to verify certain types of vulnerabilities.
But: What if we sign the form? How? Creating a hidden field with a random content stored in the Session object. If the value is submitted more than n times we just create it again. We only have to check if it matches, and if it don't just take the actions we want.
But we can do it even better: Why instead to change the value of the field, we change the name of the field randomly? Yes changing the name of the field randomly and storing it in the session object is maybe a more tricky solution, because the form is always different, and the vulnerability scanners just load it once. If we don’t get input for a field with the stored name, simply we don't process the form.
I think this can save a lot of CPU cycles. I was doing some test with the vulnerability scanners mentioned in the question and it works perfectly!
Well, thanks a lot to all of you, as a said before this solution was made with a little bit of each answer.
What I can think of is pre-populating certain form input elements based on the user's geographical information.
What are other ways can you think of to speed up user input on long application forms?
Or at least keep them focus on completing the application form?
If you have a long form, try to prune it down. Don't ask them to fill in fields that you don't really need.
If the form spans several pages, give the user some feedback as to how many more pages there are. We users hate clicking on the continue button wondering if this will be the last page.
Never lose a field that they filled in, no matter what they do. This could have security implications if passwords are involved.
Use dropdowns to provide the user with options unless there are a lot of options that the user would have to scroll through or if the terms in the dropdown aren't widely accepted (e.g. dropdown filled with Systems Engineer, Solution Developer, IT Application... I just want Programmer.).
Provide help for fields that might be hard to fill in (or provide examples).
If it is possible in your case, just collect the bare minimum up front and then allow the user to use the basic features of your service.
For the user to upgrade to a better level of service, they will need to fill in the 2nd form with more detail.
How important it is to you to collect ALL that information up front ? It is worth losing customers by demanding too much from them ? Why not demand it later at a time more convenient to the user.
Creating a multi-step wizard offering only a small number of input fields per step. Ensure that they are aware of how far they have progressed in the sequence.
The psychology is that once a user is 'invested' in a task, they are more likely to continue. If you present the whole list of input fields at once, you scare them off.
Offering musings at each step (cartoon, humor, sayings etc) makes them move to the next step out of curiosity.
Users won't mind filling in long forms if and only if they feel that the questions that you ask are important: otherwise they will be discouraged, and become impatient with it.
Remember, in a web application people have very, very short attention spans. When the user starts feeling that you are asking too much, they're usually right.
Keep required information as few as possible: other info should only be optional, and you have to give something in return to the user to compel them to complete that information.
However you implement it, please please please use some kind of Ajax hearbeat to store their progress server side and repopulate it if it's lost. There is nothing more infuriating to a user that working through a long form and having a browser or network hiccup lose their entire submission.
Whenever it happens to me I generally never give it a second shot, because at that point recreating my submission isn't worth whatever I was signing up for.
Checklist:
Explain clearly the purpose of the form. (What's in it for them?)
Prune, prune, prune, and keep questions clearly relevant!
Give the user feedback on his/her progress (if the form is split over multiple pages)
Ask for as little as you can up-front and leave the rest for later.
Clearly mark required fields
Group fields logically.
Keep labels/headings brief and easy to understand.
Prefill as much as possible - but not too much.
Spread super long forms over multiple pages and allow backtracking.
Cleverly placed "Back", "Save" and "Cancel" buttons put people's minds at ease - even when redundant.
Provide friendly (but clear!) validation error messages, in a timely manner.
Allow the user to reclaim half-filled in forms - don't lose their data!
No matter what you do, do not include a reset button. :-)
Finally:
Explicitly tell the user when the process is finished. ("Thank you! Your application has been sent.")
Tell the user what will happen next. ("A confirmation e-mail has been sent to your e-mail address, and we'll process your application within two working days.")
use Ajax to populate and update the controls asynchronously.It will speedup the filling of long application forms.
Split it up into multiple pages - there's nothing quite so discouraging as seeing that you have another 100 questions to go.
Put validation on each input and check it onblur(). If they get to the end of the page and then it says "question #2 was incorrect", chances are they've forgotten what that one was anyway and it'll be more difficult to return to it. Plus, if they answer a series of similar inputs in a particular, incorrect way, you should let them know straight away (eg: entering dates as mm/dd/yyyy when you want dd/mm/yyy)
Split the form into several steps. It's like how someone is much more likely to read five 3-sentence paragraphs than one big 15-sentence paragraph of the same length.
I agree with tim; just let them fill in the bare minimum information and then leave the rest to profile updates. If any data is necessary for the service offered on your site, ask for it when they try to avail of the service (and no earlier).
That said, I wouldn't advocate the kind of forcing function that adam suggests. It pays to give your users the warm, fuzzy feeling that they are privileged and can use ALL of the services on your site. Although, if you look at it hard enough, adam's and my suggestions are pretty much the same.
If the application needs to include a lot of information, then make sure the user can save at any point, and log off, and log in later to complete the form. This would make more sense if some of the information is not necessarily easily available. Tax returns are an obvious example, where some of the data may need to be calculated, or the user must find the relevant documentation.
In some cases the user might use the same information in multiple applications. In that case it might make sense for the user to register their details (Name, Address, Telephone numbers, etc), which are automatically filled in on each application. For example, if you had a website for a recruitment agency, they may allow users to register their details, and then to apply for a particular job, they can just include a personal statement that applies to that job in particular.
As another consideration, if some information may be incorrect (particular if this is not always clear, such as a CAPTCHA, or a user name that must be unique), either separate it from the rest of the data, or otherwise make it so a mistake doesn't mean the rest of the information must be reentered.
These are basically ways of avoiding the user having to enter the same information twice.