Sunday, December 21, 2008

Huffman coding

I've been playing with Scheme a bit. After hearing about Huffman compression in a lecture, I first wrote something in Python, but noticed that the recursion I used lent itself well to a functional language, especially where tail-calls were optimized. So I wrote this:

I've used it to construct a table for 9000 symbols with no problems, so it works pretty well. Can be downloaded here.

Saturday, November 15, 2008

ISM, an intermediate level language

For our Computer Architecture class, a partner and I created a MIPS cpu in Verilog. This was a really cool project that deserves its own description, but for now I'll focus on something I made while testing the cpu. The cpu runs MIPS assembly code. Writing assembly is not bad for simple tests, but when I started to write more complicated programs, I got tired of remembering what I was using $s0 for and so on.

So, just for fun, I quickly wrote a Python script to translate assembly-like code into assembly. The small language I designed is an intermediate between higher-level code and assembly. The syntax is more like a higher level language, assigning values with "a=b" or adding with "c = a+b". It's not a compiler, but doesn't really need to be, and with too much abstraction one would be better off writing in C anyway.

The cool part about the project is how quickly and naturally the code turned out. It only took an hour to have a fully working version that understood the opcodes, and had useful error messages for syntax errors. The code just flowed quickly as I found ways to reuse code and simplify the problem.

For example, I needed to split each line of code by tokens. I came up with a creative way to do this - Python has a built-in way to split a string based on whitespace, " 'a b c    d e'.split() " evaluates to "['a','b','c','d','e']". So in order to split on both + and - tokens and whitespace, I did "input=input.replace('+',' + ').replace('-',' - '); result=input.split()". By surrounding each occurrance of '+' with whitespace, now the string is split by both whitespace and tokens, so that a+b and a + b are both recognized as ['a','+','b']. (This could also be done by splitting on a regex, but that would be slightly more complicated).

The first, one-hour version, understands code like:
//add numbers from 1 to 25 and put result in sum.

registers i, const_one,sum,highest
seti const_one=1
seti sum=0
seti highest=25
seti i=0
LOOP:
 add i=i+const_one
 add sum=sum+i
 bne (i!=highest) goto LOOP

Then, later, I started to add more features. I added syntax shortcuts like a++, and simulated more types of branching even though our cpu only supported bne. I also added inference, where each line is checked to infer which operation you are trying to do. For example, if the line is "c=a+b" it is clear that you are doing an add instruction, so you don't have to type "add". Also, it now understands simple loops and if structures, translating them to assembly. It looks like Python if you squint.
// find prime numbers from 2 to 100.
// Place the largest found in result.
registers i,tmp, base, limit, result

// clear memory
for i,300,0
  dataWordIndex[i] = zero
endfor

limit=100
for base,2,100
  // check if we've already seen this number come up
  tmp = dataWordIndex[base]
  (tmp != zero) goto CONTINUELOOP
  
  // check off multiples of this number.
  i=base
  INNERLOOP:
    i = i + base
    dataWordIndex[i] = one
    (i < limit) goto INNERLOOP
  
  CONTINUELOOP:
endfor

// put largest prime found into result
for i,100,2
  tmp = dataWordIndex[i]
  if (tmp==zero)
    result=i
    goto DONE
  endif
endfor

DONE:

It was fun to come up with syntax for this. Branches are done with the unique, yet readable syntax "(a==b) goto Label". Array-like syntax is used for accessing data memory, like 'dataWordIndex[i] = b'. (dataWordIndex means that the index is automatically multiplied by four, so that it can be used as an array of 32 bit integers.)


When the script runs, it also prints out Verilog $display statements, which can be placed in the Verilog code to see the values of the registers.

Here is the code. ism_simple.py is the first version. I realize that it won't be useful unless you have a mips assembler and cpu.

Sunday, November 2, 2008

Rickroll Detector

Some people evidently don't understand when a meme is dead, or deserves to die. Rickrolls have been around for far too long and have even been picked up by the mainstream media. It's just not really funny anymore.

Today I was kind of bored and made this in a few hours. It's a system for checking if a link is a rickroll. You provide a URL, then some audio analysis is done, and the result is a plot showing whether or not it was a rickroll. What's nice is that it will also detect spin-offs like Barak-roll, scary roll, and so on.

How it works:
A Launchorz script directs what happens.
First, your audio is muted automatically.
Then, behind the scenes, the url is opened in Firefox. The program Total Recorder is used to capture the audio played by the webpage.
The page is closed after 15 seconds of capturing audio.
The audio is saved as a wav file in a temporary location, and your audio is unmuted.
Then, a MATLAB script is used to analyze the audio.
The audio is normalized to approximately the same volume.
Then, the cross-correlation is taken between a reference Rickroll audio and the recorded audio.
If there are peaks in the cross-correlation above around 200, then it is likely the page contained a Rickroll.

Results: After adjusting of parameters in the MATLAB code, the results are pretty good.

"Warning: rick-roll detected"


"Safe: no rick-roll detected"


Kylie Minogue's "I Should Be So Lucky".

(The mostly-evenly spaced peaks correspond with the steady beat in the music.) The results would probably be even better if the comparison was done in frequency, instead of in time. But I'm not about to waste any more time than the few hours I put into this.

Here's the script I used but it needs Launchorz, MATLAB, and the shareware program Total Recorder.

Monday, October 27, 2008

Mailman Scraper

I wrote a script to scrape messages from mailing-list archives and store them in a MySql database. It can retrieve message, author, and date information for everything sent to the list.

Code here. GPL, Requires PHP5 and mysql. You'll have to read README.txt for instructions.

Intended for use with a Mailman list, one that looks something like:

Because this just uses regular expressions to scrape the pages, it's kind of fragile. It makes some assumptions about the format of the page and might not work with other versions of mailman.

I've found this useful for a few projects. One result is a script that tracks list activity (and shows a flame-meter when the list gets intensely active; the current state is ranked from serene to mayhem). I'm also creating a site that allows searching the list archives, as well as voting on the quality of information and adding tags. Of course, one can also use this to create a Markov model of a list... but I'm not going there.

Wednesday, August 13, 2008

Documentation Viewer

This project isn't very exciting, but it will be useful to me. I wrote a documentation viewer that displays documentation information in a tree control.

I spent a lot of time going through the Python library documentation, acquiring what I thought was important, and re-organizing all of the modules into my own categories. This also required writing Python scripts to turn reStructuredText, the loose and readable format of the Python documentation, into the xml format used by the doc viewer.

I also organized all of the documentation for C.

One cool feature is that it can communicate with the SciTE code editor. So, when run from LnzScite, it appears like a tool window, and you can double-click a function in the browser, which will insert the function's name into the editor. (Watch for this in Launchorz 0.3). A planned feature is to be able to look up documentation on the currently selected function.

I wanted this program to feel small and lightweight, so I did this: the nodes are created dynamically, from the xml, when you expand a parent node. This means that the entire documentation isn't loaded into memory and no unneeded TreeNodes are created. I also spent time tuning the xml-parsing code to be very fast, using C#'s stream reader which avoids DOM overhead.

You can download the Lnz Doc Viewer here, which includes documentation for C, Python, and LnzScript (unfortunately only for Windows). The source, as usual, can be found from the project page.

As an aside, I'm starting to take advantage of some of the benefits of xml. In the source for Launchorz, there are special comments for the provided functions. Everything starting with /// is treated as a documentation string, and can be processed by a Python script into an xml file. This xml file contains the prototypes and docs for every function in lnzscript - and the same file is used by a c# program, a c++ program, and a wsh script. And it doesn't stop there - if you open the same xml file in a web browser, it nicely renders the documentation with xslt!

Friday, August 8, 2008

AI Scheme

Just for fun, a few days ago I wrote a minimax AI algorithm in Scheme. This was mainly a way to become more familiar with writing in Scheme, and so I wrote it from scratch. I had written pieces of Scheme a year ago but wanted to write something more practical. Writing in a functional language is fun because it pushes you to see problems from a new perspective. (For those unfamiliar with the concept, of the code I wrote there are no variable assignments, for loops, while loops, or C-like arrays). Also, when finished, I was able to format the code nicely.



To use this algorithm, you can provide definitions for create_new_table,get_open_spaces,place_at_position, and is_winning, and that will be all you need. This zip file contains the complete program and an example game implementation.

If I get tired of S-expression syntax, JavaScript is actually a good language for functional programming. It has the ability to do concisely express things like function() { call_something(); return function(){ do_something(); } }(); or function adder(x) { return function(y) {print('!'); return x+y}; } f = adder(1); f(1);. In fact, it did not take very much work for all of the The Little Schemer examples to be written in JavaScript; see Douglas Crockford's page and this. Python's lambda, on the other hand, is intentionally limited to containing one expression, and there not seem to be any convenient way to create an anonymous inner function with scope.

Monday, August 4, 2008

Version 0.2!

Launchorz 0.2
Documentation

I've just released version 0.2 of Launchorz. It comes with significantly more features, and even a dedicated code editor. If you're using Windows, it will save you a lot of time. Try it!

The code editor is a custom build of Scite. I've added autocomplete and tooltips to make it faster to write scripts.

See screencasts here

Download it here!

Thursday, July 10, 2008

Launchorz

Release!

For the past few weeks I have been working on a project for Windows automation. The result is Launchorz, Powerful JavaScript libraries for Win32 automation.

Look at this!
Process.open('notepad.exe')
Mouse.move(1,1);
Time.sleep(500);
Keyboard.send('hello!')
Dialog.askYesNo('Question','Did you expect this to happen?');


LnzScript is the language of the Launchorz project. It is JavaScript (run locally, not in a web browser) with libraries for simulating keystrokes, mouse movements, working with files, and a lot more.

Read the actually substantial documentation:
http://downpoured.github.com/lnz/documentation.xml

Try it now! Download and get the source from Github:
http://downpoured.github.com/lnz/

Look at what it can do:
Screencasts

One cool aspect of the project: documentation is written inline with the C++ code. Then, a Python script creates an xml file from this documentation. This same xml file is used for reading by C++, C#, and even rendering to the browser! I wrote an xsl transform that renders the documentation, so that it can be seen in a browser. If you look at the source, you'll notice that it is xml.

AutoIt exists and is good at what it does. However, its syntax and minimal data structures will not do for non-trivial programs. More of my rational is given in the documentation.

Leave a comment, or send me an e-mail if you are interested in joining the project. Please try it and give feedback - I will consider feature requests.

Sunday, July 6, 2008

Blinkbeat

Here is another of my audio experiments, called Blinkbeat. It uses the Caps/Num/Scroll lock lights on your keyboard to blink along with the song you are playing! Click on the picture to see a video (Youtube):

The FFT is used to find the energy at different frequencies. What I display is kind of like a histogram of the frequencies.

I have also added a beat detection option, which is not finished but seems to do a decent job. For beat detection, I divide the song into short segments, and use the FFT to find the low-frequency energy in each segment. So, I now have another signal in time, which is the low-frequency-energy over time. Finally, I take a second FFT of this new signal, which is finding the frequencies in this signal. I estimate the rhythm of the song to be the peak in these results, and return the result in BPM.

Downloads

Windows binary

Check out the source at GitHub, downpoured.

Known issues: The normalization of lengths isn't perfect. If you try to play a long clip, the lights and the audio will eventually lose sync. Also, I draw the bars by changing the dimensions of several button controls :).

WaveAudio Developments

I've also been working on the WaveAudio sound library in C#. It's pretty nice to be able to write:

Sine note = new Sine(440.0);
WaveAudio wave = note.CreateAudio(1.0); //1 second
wave.Amplify(1.5);
AudioPlayer pl = new AudioPlayer();
pl.play(wave); // and now you hear it!
I spent a lot of time yesterday writing some really sweet code for synthesizing audio. By the end of the day I found out that what I had sounded a lot like the cheesy "choir" effect on old keyboards!! Still, this is pretty cool, and there are many more parameters to tweak. I'm owe thanks to Wikipedia and Wikibooks for giving me ideas.

I added an algorithm for autocorrelation pitch detection, which works pretty well. It can tell what pitch I'm singing, and works on various instruments as well.

Friday, June 27, 2008

Audio Experiments!

This program is a little interactive demo of 3 "audio experiments" I came up with. (An example of its sounds). My goal here was to create interesting sounds, experiment with FFTs, and test my digital audio code - but more on that later.

First, there is the "Vibrato Lab". A quick overview: Wav files store a sound as a long list of numbers, representing air pressure vs. time. If the computer plays through this list of samples at a faster rate, we hear a higher pitch. For example, playing every other sample results in something sounding an octave higher and twice as fast. If the samples are played back at a varying rate, the pitch of the output varies, making a comical wavy sound. This is how I wrote a vibrato effect.

When testing the vibrato effect, I found that it could sound interesting if the vibrato effect were applied twice - something like vibrato(vibrato(sound)). So, this is what the vibrato lab does. Use the sliders to change the width (how widely the pitch varies) and the frequency of oscillation. Use the checkboxes to enable another vibrato effect. Play with it!

Next, the "CrFeedback". Have you ever played with video feedback, where when you tilt the camera the picture spirals and twists? When experimenting with audio code, I invented an effect that I like to think of this as the audio analog of those spirals. In this case, the song spirals out of control and degenerates into interesting noise. What you hear is like the sum of the song and many slower versions of itself. Play with the parameters - the "right" settings seem to be dependent on the clip you choose.



Finally, "Frequency Sketch". Last semester I took a course in Signals and Systems. Fourier transforms and the like are very cool, and the math was interesting. Most of the course dealt with signals in the context of voltage and current, but I wanted to apply what I had learned to audio. I wrote fourier transform code, and played around with discrete fourier transforms of audio significantly. As the name implies, the FFT is a faster algorithm for this, and so I replaced my fourier code with some FFT code by Don Cross, originally in C. Anyways, in this experiment, you can draw in the frequency domain, and hear the result! I've wanted something like this for a while, and there is still much left to study.

Microsoft Paint is the user interface for Frequency Sketch :). If you draw red pixels or lines on "input.png", you can hear the results. For example, drawing one red dot creates one sine wave, and so you will hear one pitch. The axes in blue show Hz. Note that the highest red pixel is chosen for a given frequency if there are more than one red pixels in a column. So far I've created ambient, flute-like, and eerie siren effects, but this is just the beginning.



All of this was accomplished in a few days - using a library of digital audio code I've been working on for the past few weeks. I created a project in C# called CsWaveAudio; features include saving and opening wave files, synthesizing sounds, FFT, and certain effects. One interesting part of the library is that it stores samples as doubles, between -1.0 and 1.0. This makes it much easier to do calculations - and when writing effects you don't need to consider whether you are in 8-bit or 16-bit. The library also includes code for some "instruments" made from adding sine waves.

Downloads


Windows binary

Examples of sounds

CsWaveAudio, a C# digital audio library, GPL

And, get all of the code: Downpoured at Github

More notes: in the vibrato lab, the "tremolo" option creates oscillation in amplitude, not in pitch. Also, try clicking numbers to see what happens. The code for the CrFeedback effect is in the readme file - check out the svn source to modify it. There is a "randomize phases" checkbox in frequency sketch and I'm still not sure why this makes such a difference. Beat frequencies can be created because of the spacing between pixels in the input image. Maybe I should look into interpolation.

Monday, June 23, 2008

LaTeXinWord

When I have time, I type up notes from school in LaTeX, so that they will be archived and legible for the future. This works ok, but I use many quick illustrations and diagrams, which are a pain to add to a LaTeX document (or at least are in any editor I've used). Also, even when the picture is saved, and in the right format, you can't really control positioning. By design, LaTeX has its own ideas about where it thinks your figure should go. This is good for actual "figures" as you would find in a paper, but I want my little diagrams to be inline with the text and not on the next page. In Word, by contrast, I can select an image and Ctrl C, Ctrl V it right into the document, but LaTeX's formatting of formulas is so much better.

As it turns out, I can have a combination of the two. There is a good open source project, LatexInWord which I recently discovered. Now I can keep the formulas I was using but have more control over layout of pictures. The interface is good - all one needs to do is press Alt-L to insert Latex. Also, I can have this open side-by-side with Lyx in view source mode, or another editor, and copy and paste. (There appear to be similar macros for OpenOffice, like this.)

Of course, I'll still write my actual papers in standard LaTeX, but for my personal notes, this should work well.

Tonight I wrote a quick macro to be used with LatexInWord. It prints out the LaTeX source for all of the formulas in the document. I thought this would be useful to have for archival purposes (in the future I might not have LatexInWord or Word so a plain text copy is good). This is basically my first Word vba script, but it worked when I tested it.
Sub ExportLatexAsText()
    Dim strRes As String
    Dim i As Integer
    Dim newline As String
    newline = Chr(13) & Chr(10)
    i = 1
    For Each currentShape In ActiveDocument.InlineShapes
        If currentShape.AlternativeText <> "" Then
            strRes = strRes & newline & newline & "Formula " & i & newline
            strRes = strRes & currentShape.AlternativeText
            i = i + 1
        End If
    Next
    Selection.Text = strRes

End Sub


You can add this to the document under Macros. Just start editing an existing macro, and when you're looking at the source, put this subroutine in.

Friday, June 20, 2008

Googlecalc

This has been done before several times, but usually in Perl. After reading a version by Matt Sparks, I quickly wrote a C# version. I think this is something that will be useful. You type in an expression, which is sent to Google, and the result is printed back.

Bin (Windows exe) Source, project file is Vs2008 but the source should work in 2005.

On another note, while I have enjoyed reading entries in the Obfuscated C Code Contest for a while, there is another interesting contest that sounds just as fun. The Underhanded C Contest is a "programming contest to turn out code that is malicious, but passes a rigorous inspection, and looks like an honest mistake."

Another clever contest is the Obfuscated V contest in 2004, where contestants wrote short programs that appear to simply count votes, but actually influence the results.

Thursday, June 19, 2008

Summer

I have been writing code, but that will have to wait. I just needed to post something since it's been a while.

In Java, Math.abs can return a negative number (!). I heard about it from here.

Python 2.6 is in beta. In Python, last fall Guido talked about the GIL. Now I am interested to see a "multiprocessing" module which "supports the spawning of processes using a similar API of the threading module". This is pretty cool - it will allow programs to take advantage of multiple processors, while maintaining the GIL.

A Python 3k hack from here. At least on the Python 3k I'm using , __signature__ isn't available yet, so this doesn't work yet. Annotations can be useful though.
#posted by Nick Coghlan
#on Python-3000 list,
#http://mail.python.org/pipermail/python-3000/2006-May/002033.html

#Apparently __signature__ hasn't been added yet. This won't work yet.

class Annotate(object):
     def __init__(*args, **kwds):
         self, args = args[0], args[1:]
         self.arg_notes = args
         self.kwd_notes = kwds
         self.return_note = None
         self.strict = False

     @classmethod
     def strict(*args, **kwds):
         cls, args = args[0], args[1:]
         self = cls(*args, **kwds)
         self.strict = True
         return self

     def returns(self, note):
         self.return_note = note
         return self

     def __call__(self, func):
         func.__signature__.update_annotations(self)
         return func

@Annotate(str, str, int).returns(int)
def f(a, b, c=0):
     # some operation producing an int. . .

@Annotate.strict(str, str, int).returns(int)
def f(a, b, c=0):
     # some operation producing an int. . .


I am currently working for Avid in Tewksbury, MA, as a "Media Engine Infrastructure Software Engineer Intern". It's going well so far as I dig deeper and deeper into c++.

Monday, April 28, 2008

Consonance

When two notes of close frequency are played simultaneously, one can hear what is known as a beat frequency. Say two violinists are playing. One plays an A, 440Hz and the other is playing slightly sharp at 445Hz. One hears a 5Hz "beat" buzzing, that arises from the sum of the two sine waves.

When the two tones are very close, the sound is no longer unpleasant, and the result sounds like a "smooth", flowing note. When the tones are less close, one hears the "buzzing", vibrating beats. When the two tones are significantly separated, one simply hears two notes. In the 19th century, the physicist Helmholtz posed a theory: what if humans dislike the sound of beats, and that the more the beats are noticed, the more "rough" the sound is perceived. He developed a "roughness" curve based on some experiments:
(lambda*x^2)/((33^2+x^2)^2)

Now, if we go back to the example of the two violins, we can predict how pleasant the two sounds will be to each other. This gets interesting, because a violin plays more than the fundamental note; harmonics are played also. So if the first violin plays an A, and the other plays a slightly offtune E, there will be roughness between the 3rd harmonic of the A (E'), and the 2nd harmonic of the E (E'). If the roughness for each interaction for 6 harmonics is summed, an interesting pattern emerges:


This figure plots the sum of roughness for 6 harmonics, when one plays a fixed note, and the other violin changes pitch. The peaks are areas of high roughness, and the valleys are lower roughness. The notches in this plot line up perfectly with the intervals we use in music! The central valley is the interval of a perfect fifth. The other dips are other familiar intervals. This offers an explanation of why certain intervals are more "consonant" than others, and more. In developing temperament systems, it was found that some approximations of notes are more tolarable than others. For example, we are very perceptive to a slightly-off 5th, but not to an adjusted 3rd. This might correspond graphically with the "steeper" valley of the 5th. Also, the shallower valleys reasonably match with what we consider less consonant intervals.

Using this, Helmholtz was able to explain quantitatively why a major 3rd played by a clarinet and oboe sounds better when the clarinet plays the lower tone (the clarinet sounds only odd harmonics).

The figure is a screenshot from a program I wrote to explore this. When you click on the plot, you hear the interval. It also includes a modern model of roughness, Vassilakis (2007).

Download (Windows, .NET 2.0) Source (including Matlab plots)


But do beats and perceived roughness cause dissonance, or are they just correlated with dissonance? Helmholtz's theory does not explain everything. Even when beats are eliminated, the intervals sound dissonant. The notes of our scale are also found in the harmonic series. One could alternatively say that notes are consonant when their harmonics match. This paper notes many criticisms of Helmholtz and the theory of roughness. I will have to study this further.

References: "On the Sensations of Tone as a Physiological basis for the Theory of Music" by Hermann Helmholtz, translated by Ellis
Roughness Calculation Model by Vassilakis, 2007.
Sensory Theories of Musical Consonance

Saturday, April 26, 2008

Fun with Real-time Audio

Over the past week, I've had fun with real-time audio. I'd worked a bit with sound in the past, but in Python, where synthesizing audio is noticeably slow. This time, I coded in C#, which is a lot faster and has access to DirectX. I use DirectSound to put raw bytes of data right into a buffer, enabling seamless audio.

After experimentation, I came up with a program that plays a continous tone. While it is playing, you can change the pulse-width and "smoothness", creating interesting sounds. Quickly changing pulse-width creates the illusion of motion. The "smoothness" parameter goes from sine wave-like to a square wave (approximated by quadratics). As another experiment, I added a joystick interface - if you plug in a game controller and open the program, you can use the joystick to control these parameters. I encourage the reader to think of more audio effects and interfaces.



I wrote a custom control plotting the waveform, which was useful. The program also contains my implementations of waves such as square, triangle, sawtooth, white noise, red noise.

As an aside, delegates in C# are very convenient, allowing you to easily store and pass refrences to methods. What is great is that there is so little code involved. Here's an example from my sound class:

// Takes integer index, returns double between -1 and 1
delegate double SoundDelegate(int i);
...
double timeScale = frequency * 2 * Math.PI / (double) 44100;
SoundDelegate fn = delegate(int i)
{ 
    return Math.Sin(i * timeScale);
};
return create16bit_sound(fn);
Notice how the anonymous function can even keep references to variables. It's better than a Python lambda.

Here is a sample. The real thing sounds even better, the rate of updating was limited in the process of recording this sample.

I found that smooth pitch-changing for real-time audio could be complicated. If you just change pitch, you hear a lot of "clicks" - the period doesn't line up with the size of the buffer, causing a click every refresh. I sidestepped this problem by only playing frequencies that fit into the buffer, but that's not a very good solution. If I took the previous phase into account, I could fix this.

Download (Windows, .NET 2.0, DirectX9) Joystick is optional
Source GPLv3

If you want to write DirectC audio in C#,the following may help: To get the Managed DirectX dlls, I downloaded the "DirectX 9.0c Redistributable for Software Developers", ran it, and opened the resulting ManagedDx.cab file. Also, using these in Visual Studio 2005, I would get an exception upon starting the program. One can disable this false excption in Debug->Exceptions->Managed Debugging Assistants, uncheck LoaderLock.

Tuesday, April 15, 2008

Maurader's Map Update

Over 1/4th of the students at Olin are on the Maurader's Map!

We improved the location algorithm. It is successfully at room-level precision, which is really cool, and the accuracy is very good. As you can see, we've added the ability to have custom icons. The client runs on Windows, Linux, and Mac now.

And, you can actually use it to find someone. Now we just have to persuade the professors to use it.

Sunday, March 30, 2008

gedit Regular Expression Plugin


A friend told me that gedit was missing a regular expressions plugin that could replace. I use gedit occasionally, and so I worked on this plugin. It uses Python's good regular expression module, and supports backreferences, making it possible to do significant text processing.

This plugin is based on extensão para o gEdit by Daniel Carvalho. I fixed bugs, cleaned up the interface, moved the menuitem to the Search menu, added backreferences support, added replace all, and added an option for case-sensitivity. Also, the search mode is multiline, so ^ matches the start of a line. Download
To install, place the files in ~/.gnome2/gedit/plugins/. Then open gedit, choose Preferences from the Edit menu. On the plugins tab, you should see Regular Expression Replace in the list. Check to enable. Now, you should have a "Regular Expression" item in the Search menu. Tell me if you have any feedback.

Backreferences are very useful. In the screenshot above, I've written a regular expression for turning "one.tim" into "tim.one", and so on.

Saturday, March 22, 2008

RenameCells

Renaming files in Windows Explorer is somewhat slow - select, F2, type a new name, select the next file, and repeat. I thought it would be cool to instead use a spreadsheet interface - where pressing Enter drops you down to the next line.

So, I wrote RenameCells, a Python program for quick renaming. I also added features for batch renaming like Rename by Pattern, Replace, and Regular Expressions. You can see all of the changes in a grid before you press Rename. The grid interface also lets you copy and paste from Excel or another program. See the tutorial for information about all of this.

Update! In 2012, I moved this onto Github!

Tutorial explaining use.
Download for Windows (just unzip it and run RenameCells.exe)
Python source, cross-platform, requires wxPython.

Update -- this is now on GitHub, and I use it frequently: Downpoured at Github.

Saturday, March 8, 2008

Pythonpixels

A few days ago, I wrote Pythonpixels, an original interface for image processing. (Download).

Let's say you want to quickly prototype a new imaging algorithm. As a simple example, you are increasing the Red channel of an image by 40. Typically, this would first involve quite a bit of code, in order to gain pixel-level access to the image. Also, once testing your project, you would need to set up a test framework and enter the debug/recompile cycle.

With Pythonpixels, this whole process is as easy as typing
map:
    R=r+40
Lower-case "r" means the old red value, and "R" means the new red value. That's all there is to it. To test the effect, type this into a box, press Run, and you see the results instantly.

If you wanted to create a gradient,
loop:
    R=x+y
    G=x-y
    B=0
(It's fun to experiment making patterns this way. In the examples that come with Pythonpixels, I played around with sin and cos as well to make funky patterns.)

Also, what you type is interpretted as Python code. (Any value > 255 or < 0 is truncated for you). So, you can include complicated logic, and read the output from print statements.

If you want to do something more advanced, the entire image is exposed as imgInput. The following example makes your image all wavy.
from math import sin, cos, atan2, sqrt, pi
midx = width/2
midy = height/2
nwave = 12
imgOutput = ImageChops.duplicate(imgInput)
imgInputArray = imgInput.load()
imgOutputArray = imgOutput.load()
twopiconst = 2.0 * pi / 128.0
loop:
    newx = x + nwave*sin( y * twopiconst)
    newy = y + nwave*cos( x * twopiconst)
    if (newy>0 and newy<height and newx>0 and newx<width):
        imgOutputArray[x,y] = imgInputArray[newx,newy]


Or hue shift:
loop_hsl:
    H=h+0.2


It comes with other effects, including convolution matrices, fractals, kaleidoscope, and posterize. Data-oriented operations work very well, too, like finding pixel values, statistics, or making a histogram. Other features include pasting an image from the clipboard and batch-processing a folder of images.

Try it out! Windows Download (3.7Mb, GPL). Unzip the file and run Pythonpixels.exe. Let me know what you think, and show me scripts you come up with.

Source Python 2.5 (GPL). Requires packages tkinter, python-imaging (PIL), python-imaging-tkinter (ImageTK).

You can check out the source. 2013 Edit: moved to Downpoured at GitHub!



More details: The keyword "loop" loops through every pixel, (providing the variables x and y), and if you assign to R, G, or B, the output value at that position will be changed.

The keyword "map" sets up a precomputed table for red, green, and blue values. This makes the effect very fast, but limits what can be done - the effect cannot depend on x,y position, and hsv or hsl cannot be used.

"loop_hsv" and "loop_hsl" work in the same way as loop but are in different color spaces.

Tuesday, March 4, 2008

Maurader's Map - In Use

We've released a beta of the map, and people are starting to use it. (That's a real screenshot of the people online right now.)

Saturday, February 23, 2008

Unplot

Sometimes I want to "un-plot" data. I find myself in the situation where I have a plot of the data, but not the actual coordinates of the data itself. So, today I wrote this little tool for turning a bitmap image into the x and y coordinates of the curve. (Some Photoshop work may be necessary for cleaning out other lines in the image). It simply takes the image, starts at the top, and scans downwards until it hits the curve, and then works from left to right. The data can be saved as CSV, to be easily loaded into Excel or Matlab.


I've tested this on figures from research papers, and it works. This will be useful for recovering data from a figure when the original data has been lost. Also, I can use it to get audio data from Audacity.

Watch this screencast of it in action. This will show how to use it.

And here is the program (897K). Currently Windows only, but I can port it to Linux if there is interest.

WordMash

In high school, I wrote a short JavaScript program to create Markov chains while waiting in an airport. The idea is simple but yields incredible results. First, you enter some lengthy text. A model is created by keeping track of all of the words that follow a given word. If there is a fairly large input text (Sherlock Holmes, or the Times), the slight amount of context will be enough to create sentences that (almost) make sense:

I told you yesterday, and I trust, with you, however, as I spoke, and it is clear and concise.


However, the randomness produces results that are often ridiculous!

Mr. Gordon, a retired Marine lieutenant general and a metal-schooled three-ax attack into songs that are both action-packed and gratuitously stylized. "Where Can I Stab Myself in the course of research for gut-level emotional impact," sheriff's officers said.


I recently made a site called WordMash where you can try it yourself, with texts like The Adventures of Sherlock Holmes. Best of all, you can paste your own text in, and see what happens. The code is all GPL, so have fun with it. Leave a comment if you come up with something good.

Tuesday, February 12, 2008

3 Windows Utilities

First, a rich text-to HTML converter. You can use it alongside WordPad to get a ghetto Dreamweaver.

I was looking for a quick way to convert some RTF documents to HTML. Parsing the RTF was too much effort. Pasting into Dreamweaver seemed to strip formatting. I then thought of an idea that would make this very easy. You can paste rich text into a "ContentEditable" element in Internet Explorer, and it will retain much of its formatting - and then getting the innerHTML of that element produces the HTML!

Bold/italic/underline, fonts, colors, indentation, and even tables(!) are supported. I threw together a demo here. (Requires Internet Explorer). Also of note is this cross-browser rich text editor that also works in Mozilla using designMode.

Next, a small tool for finding colors. You know the eyedropper tool in Photoshop that can pick up a color from an image? This will do something similar for anything visible on your screen. It will grab the RGB value of the color just underneath your cursor, in any Windows program.

Download it here (zip, no installer). Run the program and press the space bar to update the color to be whatever color is beneath the mouse. The idea came from altAnswerColor from Altuit.

Finally, I use Microsoft Paint occasionally to draw simple lines and diagrams. To get arrows, I'll have to copy screenshots from Word or something. I wrote this small program to have quicker access to arrows.


When you click an arrow, it places an image on the clipboard that can be quickly pasted into Paint. Simple, yet useful.