I have such log format:
[26830431.7966868][4][0.013590574264526367][30398][api][1374829886.320353][init]
GET /foo
{"controller"=>"foo", "action"=>"index"}
[26830431.7966868][666][2.1876697540283203][30398][api][1374829888.4944339][request_end]
200 OK
The entry is constracted using such pattern:
[request_id][user_id][time_from_request_started][process_id][app][timestamp][tagline]
payload
Durring request I have many point where I log something - app basically has complex behaviour. This helps me debug a lot the user behaviour.
The way I would like to parse it is that I would like to make have directory structure like this:
req_id
|
|----[time_from_request_started][process_id][timestamp][tagline]
|
etc
Basically each directory will have name based on req_id, with files wchich names are rest of tagline. These files will include payload.
And also I will have other directory, with users ids, which will contain symlinks to request done by this user.
First question: Is this structure correct? In my opinion it will make easy fast log access. The reason I want to use directories and files is that I like unix approach, and try it (feel by myself its drawbacks and advantages)
Second question: I will have no problem to use ruby for creating this. But I would like to learn some new tool, which is better suited for this. I am thinking about using just unix tools (pipe, awk etc) to achieve this, or write parser in golang which I am learning right now (even have time to implement simple map reduce). What tool is best suited for this?
I would not store logs in a directory to see how the users behave.
Depending on what behaviour you want to keep track of you could use different tools. One of these could be mixpanel or keen.io.
Instead of logging what the user did in a log file you would sent an event to either of those (they are pretty similar, pick the one you think has better docs / lib), then you would graph those events to better understand the behaviour of your users. I've done this a lot recently, to display data in a nice way I've used rickshaw.
The key point why I'm suggesting this is that if you go the file route you will still have to find a way to understand your data, something that graphs will help you a lot at. Also, visualization is something keen.io does by default, you may still want to do your graphs but it's a good start.
Hope this helped.
Is this structure correct?
Only you can know that, it depends directly on how the data needs be accessed and used.
What tool is best suited for this?
You could probably use UNIX tools to achieve this but it may as well be a good exercise to practice your Go skills by writing this. It would also be more extensible.
Does Trac provide a way to automate generation of change logs from a group of tickets? I'm interesting in giving a list of completed tickets to someone with access to Trac, preferably in a human readable format, something like a word doc or plaintext.
If Trac doesn't provide this functionality directly is there an external tool I can use?
Note that I am aware of the question How to generate changelog from Trac and it doesn't help me.
As you must have already noticed by now is, there's no dedicated function in Trac for changelog creation.
So you'll want to use the report/query interface, that certainly can harvest ticket data and has grouping/summarizing capability for a changelog-like report. Depending on your specific needs this could be enough, if you take some time to customize. Ultimately you need to provide more details on your needs, or this question is too unclear to hope for a satisfying answer.
Note too, that for Trac itself there's a dedicated wiki page with more than what ticket data alone can deliver.
How much information do you need in this "list of completed tickets"? One thing I've done before is create a report that shows the desired information and then "print" the page to a PDF file using one of the many PDF-creation utilities available. You can also use a tool like wget to grab the results in HTML format from a script.
I'm trying to decide on a GTD app. Does anyone know of one that automatically syncs with Trac or, better yet, FogBugz?
My suspicion is that none does. Which leaves me with writing a script that does it for me.
Things stores its data in XML, but the contents of the tags are all binary, which makes writing a script nigh impossible.
OmniFocus stores its data in XML, and the contents are literal text. Plugin or script is possible.
The Hit List stores its data in a sqlite3 database. Possibly easier than XML, but I'm not sure yet. The downside is that THL doesn't support recurring tasks, which makes it less useful as a GTD app.
Has anyone tried this? Have I missed an obvious app?
ThinkingRock - Java application, XML data format with plain text, supports recurring tasks. No automatic integration built yet that I know of, but another possible option to script for.
Tomboy has some level of Bugzilla integration but nothing complex. Alternatively it would be fairly trivial to sync something plaintext based such as Vimoutliner (IMO: possibly the best GTD application ever) or Taskpaper.
Probably in terms of easiness it would go:
plaintext > XML > Database > Binary format X
You could just use wget and/or a simple perl script to download the tasks then run a few regular expressions to get it formatted correctly e.g.
<li> ... </li> -> [ ] ...
or in code:
s!<li>(.*)</li>![ ] $1!g
Why not use the task features of the bug tracking systems you're looking at as your GTD tool? Also have you looked at (task coach)[http://en.wikipedia.org/wiki/Task_Coach] It stores all its info in XML.
Recently I've been doing lots of weekend coding, and have began to really need a bugtracker as things are gaining speed. This is probably the worst case scenario because I basically have to let things cool down over the week,so I simply can't remember the bugs in my head. So far I've been using a text file to jot down bugs,but I'd rather use something a bit better.
The biggest points here are ease of use and very little setup time.Don't want to spend more than an hour learning the basics and trying to install something. Also in my case I'm on a Mac so that would help, but solutions for other platforms are welcomed as they will likely help others.
FogBugz has a student/startup edition that's free indefinitely, for 2 or less users.
Personally, I use Excel. (Wait, come back, I'm not crazy!) For a bigger / team project, I've gotten a ton of mileage out of Bugzilla, but that tends to be kind of overkill for a one-person project.
But, a well-organized spreadsheet, with columns for things like "status", "description", "code module", "resolved date," etc, gets you pretty close to what you'd need for a small project. Sorting a spreadsheet by column isn't anywhere near a search, but its a whole lot better than "find in text file."
Heck, if you use Google docs rather than excel, you can even publish the thing as an RSS feed and get it anywhere.
And, the major advantage is that the setup time and learning curve are both effectively nil.
Addendum: And of course, the instant your "One-Person Bug Tracker" becomes a "Two-Person Bug Tracker" you must switch to something better. Bugzilla, FogBugz, anything. Trust me, I've been there.
Trac or Redmine are both pretty good. I don't know how easy they are to set up on a Mac.
It's worth mentioning that FogBugz also has a free version for up to 2 users, which would suit you. It is hosted so there is no installation and you can use something like Fluid to access it in its own window.
I don't think you need a full blown bugtracker for your scenario.
Try tiddly wiki, store each bug in a tiddler and give them tags like 'open' or 'closed'.
There is no installation required (only one html file), and it's very easy to use.
And platform neutral.
If you're working on a LAMPP stack, then for ease of setup and use I would probably recommend Mantis. It's written in PHP / MySQL and the only installation involved was specifying where the database should be created and what credentials should be used.
Oh, and its FOSS.
I would suggest Omnigroup's Omnifocus - it's an excellent task tracker, and if you just make the mental leap from bug to task, I think it works famously for one man projects as well as being an excellent way to organize your no doubt burgeoning task queue.
Eclipse has a really interesting system--I don't know why so few people seem to know about it.
It's tied in with their to-do list. It gives you the ability to enter bugs with as much or as little info as you like. You can tie it to versioning or an external bug tracker if you like. It's a decent bug tracker in itself.
The real trick is how it works with your source code.
Before you begin work you select a bug from the list. All the time you're coding, it tracks what files you are editing. It can close old tabs for you, and will also highlight areas of the source tree that you have modified a lot.
The nice thing is, you can go back to any bug you've edited an you will get your "Environment" back. Not only all your notes and stuff, but the same tabs will open up and the same sections of code in the navigator will be highlighted.
Also eclipse works with virtually any language, it's not just restricted to Java...
let me put in a good word for ditz - it's a bit bare-bones, but it has the invaluable feature that bugs are checked into your repository. it's also very easy to use once you get used to its way of doing things
You can use fogbugz for free if you're a one man team.
It's super easy to use and quick to learn.
They made it so that bugs are really easy to enter, no mandatory fields.
I'm the author of BugTracker.NET mentioned in another post. If I were looking for a tracker for JUST ONE PERSON with MINIMUM hassle, I'd use FogBugz, because it's hosted. No installation, no need to worry about backups.
But, what are you doing about version control? Don't you have to worry about that too, and backing that up? If so, consider something like Unfuddle or CVSDude where you can get BOTH Subversion and Trac, or Subversion and Fogbugz.
I use Mantis at home and I'm happy with it. It can be a pain in the arse to get it working so you can choose to download a free and ready-made VM installation. Cannot be easier than that,
Maybe a spreadsheet would be the next logical step? I know it sounds really un-sexy, but if you're the only user, you don't have to worry much about others mucking it up, and it adds a few basic features over a text file like sorting. Then if you later need to graduate to something RDBMS-backed, you would likely have a feasible import path. I just know that for me, when working by myself, I don't tend to get around to putting bugs in anything that requires more care and feeding than that (of course when working with others the collaborative needs make a more defined repository a requirement, but that's a different story).
EDIT: After noting the availability of free, hosted access to FogBugz, I'm re-thinking the bar for care and feeding...
RT from BestPractical is great.
I also get a lot of mileage out of just keeping a list of items in a text file with vi, if I can express them all in one line. This is usually for many small todo items on a single component or task.
I've tried bugtracker.net and even though it's a little bit rough on the edges, it's free and was built with ASP.NET:
http://sourceforge.net/project/showfiles.php?group_id=66812
Are you using a source control repository as well? If not, you really should, even though you're only a one-man team.
My personal preference is to use a VMWare Virutal Application (free) that offers no-hassle setup gives you access to both Trac and Subversion. You can find many different virual appliances through searching. Here is one example of getting a Trac/SVN virtual appliance up and running:
http://www.rungeek.com/blog/archives/how-to-setup-svn-and-trac-with-a-virtual-appliance/
Trac is an excellent project management tool that sports a bug tracker, wiki, and integrated source control management. It's adaptable to your needs, and fits me very well personally.
I use bugzilla for this purpose. Plus for me was that it has integration with Eclipse (precisely with Mylyn). FogBuzz has it to but AFAIK it is nonfree.
Plus it sits on my laptop so I can code and add/remove bugs when offline (it was biggest disadvantage of hosted solutions for me)
Installation was not a problem in Ubuntu (and any debian-based distro I suppose).
I dig ELOG in those cases, it's more of a personal blog, but it's easy to handle and install, the data is local on your computer and you can search all entries via fulltext. Always sufficed for me.
If you have a Windows box with IIS and MSSQL (including SQL Server Express), you should look at Bugtracker.net. It is free and open source (you get the source code), and it is extensible.
Even if you are a one man shop, having a free bug tracking system with this much power will allow you to grow over time, because it is fairly easy to add future users into the system.
You can also customize it for the look of your organization, business or product.
Ontime 2008 by Axosoft is free for a single user licence. It's industrial strength and will give you alot more that just bug tracking!
http://www.axosoft.com
Jira which now has free personal licenses.
I am using leo for this purpose. To be more specific, its cleo plugin.
Of course you might need to spend some time to get used to leo, but it will pay off.
A flat text file is just a list, an Excel spreadsheet is a two-dimensional list.
leo lets you keep the data in a tree! And it also has clones.
We are developing a web application which is available in 3 languages.
There are these key-value pairs to translate everything. At this moment we use Excel (key, german, french, english) for this. But this does not work well ... if there is more than 1 person editing this file, you have no chance to automatically merge the different files.
Is there a good (and free) tool which can handle this job?
--- additional information ---
(This is a STRUTS application) But the question is how to manage these kinds of information in general (or at least in an conveinient way, which also supports multiple users editing this single file ("mergeable" filetypes))
Why not use gettext and manage separate .po files? See that blog entry.
If you can store this information in plain text then you will be able to use a version control system like subversion to help you with merging changes. Subversion is free.
The free guide (the "Red Book") to subversion gives a fairly good explanation of how this kind of merging works.
http://svnbook.red-bean.com/en/1.5/svn.basic.vsn-models.html#svn.basic.vsn-models.copy-merge
EDIT: Another thought - if you really want to stay using a spreadsheet - Google Docs supports simultaneous editing of a spreadsheet. You could import your existing spreadsheet and get your multi-user merging wishes for free with very little change to how you work.
Good Question.
There are some "Best Practice" depending on what you actually code in (java, ms-windows c#).
I solved this (but I think there must be a better way) by using a SQL db instead of excel file, and a wrote a plug for VS (VB6,........,..., emacs) that was able to insert new keys into the db without going to round trip with version control. The keys are the developers name of what they think is a best guess for a label. (key => save, sv => "spara", no => "", en => "save").
This db can then be generated as a module, class, obj, txt, to appropriate code(platform)
and can be accessed, depending on the ide, so in c#, bt,label = corelang.save;
Someone else can then do all the language stuff, and then we just update the db and rerun the generation to the platform resources.
After years of seeing localization done, including localization at large companies like Sony. I can only say the "standard" is Excel :)
There are tons of good ideas around, and probably many better ways to do it, but in real-life excel seems to be the best/cost effective solution that doesn't require training or making complex new tools to get the job done.
Found out, that Intellij Idea (at leas in version 7 and 8) has an editor for application resources. But it is not free at all. And it does not scale for bigger resource files with more than 1.000 keys.
Another good choice would be to use Google's spreadsheets ... for those who don't know it - it is like an "online Excell web-application". It can handle concurrent access from multiple users. Yay! But sadly, it comes from Google. This makes it impossible to be used in commercial projects.
So,
still searching...
cheers,
mana