I just saw a paper by Cornell reconstructing faces from sound. But I am more interested in the timbre. It might be attacked with AI, but is there an easier way? For example, is instrument a going to be on a different range than instrument b.
For the most part, instruments are going to have overlapping frequency content. IDK the specific algorithms for isolating instruments--I've heard they do exist. I would think that a big element is not just tracking all the harmonics and frequency content, but looking for correspondences in volume changes or frequency changes of the different frequencies, in order to determine which frequencies should be grouped together as a single instrument. Since instruments often play the same notes at the same time, this would be no mean feat. If you are a beginner with digital signal processing, can I recommend "The Scientists and Engineers Guide to DSP" by Steve Smith? (Free download, good book on the fundamental knowledge needed to tackle such a project.)
I am currently conducting research on American pop songs using Spotify's audio features (e.g., danceability, tempo, and valence...). But, I couldn't find any documentation that contains details about how they measured the features. I know there's a brief description of the features. But, it doesn't tell about any the exact measurement. Could you let me know where I can find it?
Thanks.
The Echonest was a music data analysis platform acquired by Spotify, and its expertise is being currently used to power up Spotify recommendation tools.
Audio Features API endpoint extracts a more "High Level" analysis from audio and songs, whereas Audio Analysis endpoint extracts more "Low Level" and technical data.
Essentially, "High-level" features are more explicit and make use of clearer semantics -plain english, in order to be easily understood by the layman ("danceability", for instance), but it all comes from Low Level analysis, really.
Here you have some documentation, if you wish to dive deeper into the matter:
http://docs.echonest.com.s3-website-us-east-1.amazonaws.com/_static/AnalyzeDocumentation.pdf
Where is the Body of Knowledge for programmers interested in developing applications that simulate traditional artists' materials and tools, such as simulating natural paints?
Is there any substantial body of knowledge or resource for software engineers interested in creating applications that reproduce the effect of painting and drawing media such as watercolor, oil, chalk, charcoal and color pencil?
Clearly the knowledge exists and is shared by software engineers at Adobe, Corel, etc. But out here in the open, where is this information?
So far I've only come across fragmentary knowledge of a little technique here or there, but have not yet found any substantial resource. If you know where I need to look, please point me there.
Where are the best academic resources? Are there any blogs that specialize in this area? Are their organizations that specialize in this?
Eureka! NPR - for those of us heretofore uninitiated - refers to Non-Photorealistic Rendering, the "area of computer graphics that makes images
resembling traditional artistic works (Mould)."
ACM Digital Library appears to be a very extensive source for research and academic papers on NPR. See here, here, and here for a good index of NPAR papers, search for 'NPR' or non-photorealistic rendering' on this page.
This page at the ACM Digital Library show an example of ways to access the material - a member (who has paid annual membership fee) can purchase the paper for $10, a non-member for $15; Alternatively, you can rent an article for such as this for $2.99 for 24 hours.
Non-photorealistic Rendering, Bruce Gooch, Amy Gooch - 2001: A preview of this book is on Google Books and it looks like a bulls-eye for subject matter expertise.
Non-Photorealistic Computer Graphics: Modeling, Rendering, and Animation (The Morgan Kaufmann Series in Computer Graphics) by Thomas Strothotte. This book looks like a goldmine. It covers things like stippling, drawing incorrect lines, drawing artistic lines, simulating painting with wet paint, simulating pencils, strokes and textures, etc.
Microsoft and Adobe have a visible presence in this field of work, and of course employ people who are prominently involved in the NPR field, and you will see their names (or that of their employers') often appear as participants and sponsors of events like the upcoming NPAR 2011 (Sponsored by adobe).
Microsoft has the impressive Project Gustav (video) - a software application dedicated exclusively to very realistically simulating traditional artistic tools such as paint, chalk, pencil, etc.
Detail Preserving Paint Modeling of 3D Brushes.
With the popularity of the Apple iPhone, the potential of the Microsoft Surface, and the sheer fluidity and innovation of the interfaces pioneered by Jeff Han of Perceptive Pixel ...
What are good examples of Graphical User Interfaces which have evolved beyond the
Windows, Icons, ( Mouse / Menu ), and Pointer paradigm ?
Are you only interested in GUIs? A lot of research has been done and continues to be done on tangible interfaces for example, which fall outside of that category (although they can include computer graphics). The User Interface Wikipedia page might be a good place to start. You might also want to explore the ACM CHI Conference. I used to know some of the people who worked on zooming interfaces; the Human Computer Interaction Lab an the University of Maryland also has a bunch of links which you may find interesting.
Lastly I will point out that a lot of innovative user interface ideas work better in demos than they do in real use. I bring that up because your example, as a couple of commenters have pointed out, might, if applied inappropriately, be tiring to use for any extended period of time. Note that light pens were, for the most part, replaced by mice. Good design sometimes goes against naive intuition (mine anyway). There is a nice rant on this topic with regard to 3d graphics on useit.com.
Technically, the interface you are looking for may be called Post-WIMP user interfaces, according to a paper of the same name by Andries van Dam. The reasons why we need other paradigms is that WIMP is not good enough, especially for some specific applications such as 3D model manipulation.
To those who think that UI research builds only cool-looking but non-practical demos, the first mouse was bulky and it took decades to be prevalent. Also Douglas Engelbart, the inventor, thought people would use both mouse and (a short form of) keyboard at the same time. This shows that even a pioneer of the field had a wrong vision about the future.
Since we are still in WIMP era, there are diverse comments on how the future will be (and most of them must be wrong.) Please search for these keywords in Google for more details.
Programming by example/demonstration
In short, in this paradigm, users show what they want to do and computer will learn new behaviors.
3D User Interfaces
I guess everybody knows and has seen many examples of this interface before. Despite a lot of hot debates on its usefulness, a part of 3D interface ongoing research has been implemented into many leading operating systems. The state of the art could be BumpTop. See also: Zooming User Interfaces
Pen-based/Sketch-based/Gesture-based Computing
Though this interface may use the same hardware setup like WIMP but, instead of point-and-click, users command through strokes which are information-richer.
Direct-touch User Interface
This is ike Microsoft's Surface or Apple's iPhone, but it doesn't have to be on tabletop. The interactive surface can be vertical, say wall, or not flat.
Tangible User Interface
This has already been mentioned in another answer. This can work well with touch surface, a set of computer vision system, or augmented reality.
Voice User Interface, Mobile computing, Wearable Computers, Ubiquitous/Pervasive Computing, Human-Robot Interaction, etc.
Further information:
Noncommand User Interface by Jakob Nielsen (1993) is another seminal paper on the topic.
If you want some theoretical concepts on GUIs, consider looking at vis, by Tuomo Valkonen. Tuomo has been extremely critical of WIMP concept for a long, he has developed ion window manager, which is one of many tiling window managers around. Tiling WMs are actually a performance win for the user when used right.
Vis is the idea of an UI which actually adapts to the needs of the particular user or his environment, including vision impairment, tactile preferences (mouse or keyboard), preferred language (to better suit right-to-left languages), preferred visual presentation (button order, mac-style or windows-style), better use of available space, corporate identity etc. The UI definition is presentation-free, the only things allowed are input/output parameters and their relationships. The layout algorithms and ergonomical constraints of the GUI itself are defined exactly once, at system level and in user's preferences. Essentially, this allows for any kind of GUI as long as the data to be shown is clearly defined. A GUI for a mobile device is equally possible as is a text terminal UI and voice interface.
How about mouse gestures?
A somewhat unknown, relatively new and highly underestimated UI feature.
They tend to have a somewhat steeper learning curve then icons because of the invisibility (if nobody tells you they exist, they stay invisible), but can be a real time saver for the more experienced user (I get real aggrevated when I have to browse without mouse gestures).
It's kind of like the hotkey for the mouse.
Sticking to GUIs puts limits on the physical properties of the hardware. Users have to be able to read a screen and respond in some way. The iPhone, for example: It's interface is the whole top surface, so physical size and the IxD are opposing factors.
Around Christmas I wrote a paper exploring the potential for a wearable BCI-controlled device. Now, I'm not suggesting we're ready to start building such devices, but the lessons learnt are valid. I found that most users liked the idea of using language as the primary interaction medium. Crucially though, all expressed concerns about ambiguity and confirmation.
The WIMP paradigm is one that relies on very precise, definite actions - usually button pressing. Additionally, as Nielsen reminds us, good feedback is essential. WIMP systems are usually pretty good at (or at least have the potential to) immediately announcing the receipt and outcome of a users actions.
To escape these paired requirements, it seems we really need to write software that users can trust. This might mean being context aware, or it might mean having some sort of structured query language based on a subset of English, or it might mean something entirely different. What it certainly means though, is that we'd be free of the desktop and finally be able to deploy a seamlessly integrated computing experience.
NUI Group people work primarily on multi-touch interfaces and you can see some nice examples of modern, more human-friendly designs (not counting the endless photo-organizing-app demos ;) ).
People are used to WIMP, the other main issue is that most of the other "Cool" interfaces require specialized hardware.
I'm not in journalism; I write software for a living.
vim!
It's definitely outside the realm of WIMP, but whether it's beyond it or way behind it is up to judgment!
I would recommend the following paper:
Jacob, R. J., Girouard, A., Hirshfield, L. M., Horn, M. S., Shaer, O., Solovey, E. T., and Zigelbaum, J. 2008. Reality-based interaction: a framework for post-WIMP interfaces. In Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy, April 05 - 10, 2008). CHI '08. ACM, New York, NY, 201-210. see DOI
How does the open-source/free software community develop drivers for products that offer no documentation?
How do you reverse engineer something?
You observe the input and output, and develop a set of rules or models that describe the operation of the object.
Example:
Let's say you want to develop a USB camera driver. The "black box" is the software driver.
Develop hooks into the OS and/or driver so you can see the inputs and outputs of the driver
Generate typical inputs, and record the outputs
Analyze the outputs and synthesize a model that describes the relationship between the input and output
Test the model - put it in place of the black box driver, and run your tests
If it does everything you need, you're done, if not rinse and repeat
Note that this is just a regular problem solving/scientific process. For instance, weather forecasters do the same thing - they observe the weather, test the current conditions against the model, which predicts what will happen over the next few days, and then compare the model's output to reality. When it doesn't match they go back and adjust the model.
This method is slightly safer (legally) than clean room reverse engineering, where someone actually decompiles the code, or disassembles the product, analyzes it thoroughly, and makes a model based on what they saw. Then the model (AND NOTHING ELSE) is passed to the developers replicating the functionality of the product. The engineer who took the original apart, however, cannot participate because he might bring copyrighted portions of the code/design and inadvertently put them in the new code.
If you never disassemble or decompile the product, though, you should be in legally safe waters - the only problem left is that of patents.
-Adam
Usually by reverse engineering the code. There might be legal issues in some countries, though.
Reverse Engineering
Reverse engineering Windows USB device drivers for the purpose of
creating compatible device drivers for Linux
Nvidia cracks down on third party driver development
This is a pretty vague question, but I would say reverse engineering. How they go about that is dependent on what kind of device it is and what is available for it. In many cases the device may have a similar core chipset to another device that can be modified to work.