Purpose

A flexible realtime processor/sequencer for audio, MIDI and other signals.

Concept

Building blocks of signal input/output and transformation, Plugins, are arranged in a signal flow Graph. Synchronous (audio) and asynchronous (control) signals are transmitted via connections set up between plugin ports (In, Out).

 

Front Matter

Introduction
  Motivation
  Concept
  Why New
  Why Free

Installation
  Software
  Hardware
  Porting
  Download
  Compiling
  First Steps
  Bugs

Implementation
  Processing Cycles
  Signal Travel
  Time
  C++ and Python
  Status

Notes
  stdmsg
  Latency
  Whoami
  System Clock
  Source Code
  Writing Plugins
  Final Word

Reference

Gtk Reference

This document presents nam, a realtime computer system for musical purposes. The project is discussed briefly in rationale, design and implementation. A reference of the Python programming interface is included that has been extracted from the docstrings of the original module.

Please understand that what you are looking at is a snapshot of a project under development. Your input is wanted to improve both this document and the project it tries to describe in general.

Newer versions can be obtained at http://quitte.de/nam.html.

Thanks

To all the fine people who have written and improved free software.

Copyleft

nam is © 2001, 2002 Tim Goetze, it is yours to the extents set forth in the GNU Public Licence.

Motivation

It is now a few years that there was a little computer that was my ticket to a world completely different from the world we experience every day. When I should have been studying, I spent my time exploring this strange and beautiful space, and whatever I did outside, in here the sky was the limit.

The computer then was an Atari 1040 ST, hooked onto it there was a keyboard that delighted me with up to eight splendid timbres sounding simultaneously, sampled twelve bits deep.

As I journeyed on exploring my alternative home in all its magnificence, it occurred to me that my powers over it were not as subtle as I would have liked them to be.

Being already acquainted with programming, I felt this to be one direction to follow.

Concept

The new system had to be universal, simple, transparent, maintainable; it had to allow virtually anything to connect, and it had to be cool about it, even if used by an intoxicated musician.

The concept chosen is a 'signal flowchart' architecture, a processing graph that is populated by various interconnected plugins. There are plugins that produce signal streams, from sources such as a MIDI keyboard, a disk file, an AD converter, an algorithm, a network socket etc. Other plugins operate on input signals and produce a transformation, and eventually there are plugins that route signal streams on to outside the system, feeding disk files, DA converters, a MIDI interface etc.

Any computer system striving for universality needs not only be modular, it also needs to have a programming interface. The demand for simplicity and friendliness made Python the programming language of choice for this task.

Why on earth Yet Another Computer Music tool?

Yes, there's a lot of packages out there that do similar things, and some of them have reached levels I can certainly only dream of. However they would not give me:

• Total control. Closed source just doesn't cut it.
   
• Verbosity. Most of the powerful packages in this field have been written by software engineers, trained to read and write code in a very mathematical style. This is a serious obstacle to anyone who is willing and able to understand the concepts, but hasn't been subject to the same kind of education.
   
• Real Time, Real Life Applications. I wanted a package that has the flexibility for applications in any kind of environment, whether it's live, studio, at home or on the beach.
   

Why is it Free Software?

Never heard of the revolution yet?   : )

Software

You will need these fine software packages:

• GNU/Linux.
  For reliable audio latencies under 90 ms, you may want to apply Andrew Morton's helpful low-latency kernel patches, or use a patched kernel package.
   
• Python,
  from www.python.org, version 2.2 or later.
   
• ALSA,
  from www.alsa-project.org. Linux kernels from 2.5 on include these audio and MIDI drivers, otherwise you need version 1.0.1 or later; installing the latest version is recommended.
   
• ladspa.h,
  the Linux Audio Developer Simple Plugin API is available from www.ladspa.org/ladspa_sdk. The file is usually installed as /usr/local/include/ladspa.h.
   
  
• FFTW,
  the Fastest Fourier Transform in the West, from fftw.org version 2, compiled to use the float type. (Soon to be replaced by FFTW3.)

• Jack,
  the linux realtime audio server from jackit.sf.net, version 0.91.1 or later.
   
• FluidSynth,
  the SF2 synthesizer from fluidynth.org, version 1.0.3 or later.
   
• GTK+ 2,
  the X interface toolkit from gtk.org and pygtk 2.0.
   

Recommended addition:

• Steve Harris' versatile set of audio DSP plugins.
   
• My own LADSPA efforts.
   
• (And of course Fons Adriaensen's, Tom Szilagyi's and all the other fine LADSPA collections.)
   

Hardware

A decent audio interface should be installed and the ALSA driver module operational. Some rare sound cards exist that will not work with this library without either the card or the package undergoing a major redesign. If your sound interface works with ALSA but not with the AlsaClock plugin after trying all sensible frames_per_cycle settings, please send a bug report.

MIDI communication is 'raw' and should work with anything that speaks the protocol and is not a block device file. The ALSA sequencer interface is also supported (without timing information or SysEx).

Porting

This package has been developed and designed to run on Linux, i386. It should compile and run on other Linux ports without modifications but don't expect it to do so right away and read on.

Ports to other unices should be straightforward, especially if they do provide a usable pthread implementation. Nevertheless, I recommend downgrading your OS to Linux, just as I always do.

The implementation works on the premise that the target architecture provides 'real' atomic integer and pointer data types, ie. these types must exist and work without the protection of a spinlock. This may rule out ports to some processor families.

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

The source package is here: nam-0.55.766.tar.gz.

This is a first, pre-pre-pre-alpha release, and some things may not work the way one would expect them to. Bug reports are welcome.

Compiling

Unpack the source tarball and change to the source directory:
 
$ tar xfz nam-0.55.766.tar.gz
$ cd nam-0.55.766

Configure the package:
 
$ ./configure.py --help
 
Configuration options are printed. You may want to choose a default build first; otherwise add options to the following command:
 
$ ./configure.py

Please understand that the configure script is not very bright and mostly trusts you have the right versions of prerequisite packages installed. In particular, it may not detect an aged ALSA version.

Build it:
 
$ make

First Steps

After building the Python extension module, you can test it using the applications from the examples directory (please read the section below first). For example,
 
$ examples/record.py test.wav
 
should record two-channel audio from the first soundcard;
 
$ examples/play.py test.wav
 
should play the recording.

Other examples are supplied to show how more meaningful applications can be built. Hack away and have fun.

If you haven't done so already, please read the notes section below about the 'stdmsg' logfile, low-latency signal processing requirements and other points of importance before installing.

To make the module available outside the source directory, issue (as root)
 
# make install

• Signal handling has not been focussed on yet; some POSIX signals may, depending on its current state, more or less badly confuse the system.
   
• In particular, sending SIGINT (pressing ^C in interactive Python mode) will confuse Python 2.2's readline module if the Gtk extension is active. You can still type at it, but it will not print anymore. Hints about what may be causing this behaviour are greatly appreciated.
   
• The WaveFile object will neither cope with all the intricacies the file standard declares legal; nor will it read and write all sample formats possibly found in a '.wav'.

Possible caveats:

• Easy on the volume. ALSA I/O has not been tested with:
  
  
  • USB audio devices
  • Non-interleaved buffer access (Hammerfall)
  • 24-bit sample sizes (most cards mask 24 as 32 anyway)
  • Non-'hw' ALSA devices ('plug', 'plughw')
  
   
• Untested on SMP systems.
   
• Untested on non-i386 architectures.
   

'Features':

• All plugins except Parts will recognize a Note event as a note-on only, causing a stuck note.
   

Bug reports can be sent to tim@quitte.de.

Processing Cycles

Signal processing and propagation is carried out in cycles. There is a cycle dedicated to synchronous audio processing, and for the scheduling of asynchronous signals there is a low-latency, high-frequency cycle (the 'Time' cycle, 1024 Hz default) and a high-latency, low-frequency cycle (dubbed 'Beat', currently cycling at eighth note intervals).

All processing cycles are run in dedicated threads of execution. Plugins that act as cycle schedulers are responsible for managing these threads; the 'Beat' cycle thread is run by the graph. If possible, appropriate scheduling priorities are assigned.

A plugin can take part in any combination of cycles. The sequence of plugin invocations in the processing cycles is determined by the graph following their layout in a two-dimensional space.

Besides their work in the cycles already mentioned, some plugins occasionally need to execute code without realtime constraints. For example a DiskStream plugin calculates it needs to read the next few seconds of audio data from a disk file. It will notify a DiskScheduler during its turn in the audio cycle. The scheduler thread is woken, and a disk processing cycle begins.

The execution of Python code is serialized by a mutual-exclusive lock; waiting for this mutex to be released is not an option under 'hard' realtime constraints. Therefore, a ScriptScheduler thread will be woken when a ScriptPlugin receives data. A script cycle is triggered, and the plugin is able to process.

Thus, the Python lock, and the free use of dynamically allocated memory by the Python interpreter make script plugins a source of potentially very high latency, limiting them to 'soft' realtime purposes.

Signal Travel

MIDI and other asynchronous signal data is transported by Events, synchronous audio by a dedicated type of AudioEvents (CyclicAudio). Event travel between an output port Out and an input port In is implemented as a lock-free first-in-first-out operation to allow different threads of processing across the connection.

A second fifo attached to every output port serves as an event cache. Besides guaranteeing that no memory allocation needs to take place under realtime constraints, this also means that events can be modified by plugins processing the output signal without touching longer-lived stored data (SysexEvents, when played through a Part, are an exception to this rule. They are sent 'as-is').

When en event has completed its journey through the graph, it is passed into another fifo, making its passage to a mediator thread. The mediator thread is responsible for the disposal of all objects, cached or uncached. Cyclic audio events, being used only within the audio thread, are immediately returned to the port they originate from.

Input ports usually support multiple connections, yielding a composite signal: audio signal is mixed, and event streams interleaved in timestamp order.

Time

Although being of floating point type, the timestamp on events relates to a tick-based system much like that common in MIDI applications. This is to make it easier to interpret and modify event timing in the context of musical time and tempo, and to allow realtime tempo changes in conjunction with prequeued events.

Plugins that care about audio signal time evaluate the 'frame' timestamp on AudioEvents that relates to the master sample clock.

The graph maintains a set of clocks reflecting the current 'tick' (musical time), 'time' (seconds) and 'frame' (sample clock). If an AudioClock plugin is present, all clocks are synchronized to the sample clock.

C++ and Python

A major goal of the implementation has been to integrate the two languages as closely as possible. The type architecture of Python 2.2, in combination with C++ templates, has made it possible to implement a C++ class family whose members are legal Python citizens.

Another aim has been to hide the complexity of a multi-threaded core from the Python API wherever possible. For example, modifications to the graph state (adding/removing a plugin, seeking etc) must be synchronized with the audio thread in order to prevent glitches or dropouts. The (quite delicate) mechanism that ensures this is working 'invisibly': Python code sees these actions are mere method calls or property assignments.

The benefits of the unification of Python and C++ classes are numerous: it is possible to subclass any of these types with Python classes, the two namespaces match very well, and the amount of code necessary to reconcile the compiled and the interpreted language can be distilled to a bare minimum. Because this Python API is considered their 'public' interface, the C++ structures can be coded in a very functional style.

This approach has already proved beneficial to the development process by enabling the author to easily set up and closely examine debugging scenarios, without the influence common debugging tools exert on a program running multi-threaded.

As a consequence of the far-going namespace symmetry, the information presented in the reference section of this document, while being extracted from the Python interface, also documents the structure and workings of the underlying C++ fairly accurately.

Yeah, sure. Send me a mail.

Status

The project currently manifests in a Python extension module, libmandala.so. This module is intended to serve as a construction kit for realtime signal processors and sequencers. The Python API is not considered stable yet.

The 'stdmsg' File

If this option has not been turned off at configuration time, the libmandala module will open a file called 'stdmsg' in the current working directory at the time it is imported. It serves as a logfile for background threads. Xruns (see below) are reported here.
The file is put into the module dict under the name 'stdmsg'.

Processing Latency

Suppose we wanted to build a realtime audio effects processor using this package. Such an application has an all-important property, that is the time it takes the signal to travel through the system, from entrance to exit.

For playing an instrument through the processor, we want this delay to be as short as possible. The parameter that controls it is (along with the sample rate) the audio cycle time, ie. the number of samples the processor will read from and write to the audio device at a time (general-purpose computer systems could not keep the pace of the sample rate were they to process one sample at a time).
With most sound cards, the value is a power of two, starting at 64 frames/cycle. At a sample rate of 44.1 kHz, one audio cycle would thus take a little less than 1.5 milliseconds to complete.

At this setting, our application will read 64 samples at every audio cycle. This signal, when read from the input, can be thought to already be 1.5 ms old. The processor applies effects magic to the chunk and sends it to an audio output. This cannot be accomplished in 0 seconds, and so another audio cycle and another 1.5 ms pass before the signal reaches the output. The total processing latency is now a little less than 3 milliseconds, not taking into account the latency AD and DA converters may introduce.

Thus, the application needs to complete every audio cycle in less than 1.5 ms, and to do so steadily every 1.5 ms. This can only be accomplished if the thread the audio cycle is run in preempts all other threads on the system, a privilege only granted to processes run by the superuser (or granted this capability by the superuser, a feature not commonly compiled into the kernel).

Even with adjusted scheduling priority, a stock linux kernel does not support the low scheduling latency needed. When the audio interface signals data is available, an audio cycle needs to start right away. An unmodified linux kernel will sometimes, especially when under severe load, take generous amounts of time until the audio thread is scheduled to run. This behaviour is improved vastly by the aforementioned kernel patches.

The sample buffers used with audio hardware are ringbuffers. This means that if an application doesn't manage to read or write quickly enough to keep up with the pace the sampling rate sets, data is lost. The read case 'overrun' and the write case 'underrun' are collectively dubbed 'xrun'.

Whoami?

As discussed above, for low-latency processing the audio thread needs root rights to run at enhanced scheduling priority.

Another requirement for realtime operation is that the process' memory is never paged out, otherwise the response to a signal may be delayed due to being forced to wait for its memory to come back. To turn off paging is also a root privilege.

For the high-speed pulse required to precisely schedule MIDI events, the device file /dev/rtc is used, which will refuse pulse frequencies above 64 Hz to anyone but the superuser.

What all this boils down to is that for ultimate responsiveness and timing precision a realtime application must be run as root. Running a Python script as root is something many computer literates will call a gaping security hole.

System Clock

The RTClock plugin and other timing mechanisms in this package rely on the gettimeofday system call. Changing the system time while working with the package may confuse the clock plugin and cause a hiccup.

Source Code

All source text is formatted for a tabsize of two spaces.

Writing New Plugins

The VCF plugin source code (file VCF.cc) provides an example of an audio plugin that is fairly simple, yet also fairly complete.

External plugins, ie. new types deriving from any of the types in libmandala.so that come in their own module, are possible. Caveat One: the plugin module must contain a module initialization function like initlibmandala(). Caveat Two: C++ exceptions thrown in a dynamically loaded module are not caught by the libmandala module (at least not if the package and the new module have been compiled by gcc 2.95.4).

Final Word

It's not the power you have, it's the use you put it to.

Object
 + AudioSignal
 |  + CyclicAudio
 |  + DecycledAudio
 + Event
 |  + FloatEvent
 |  + MidiEvent
 |  |  + Note
 |  |  + SysexEvent
 |  + TempoEvent
 |  + TextEvent
 |  + TimeEvent
 + NamedObject
 |  + EventList
 |  + FFT
 |  + GUSVoice
 |  + Granular
 |  + Graph
 |  + In
 |  |  + LadspaIn
 |  + OSS
 |  + ObjectFifo
 |  |  + EventCache
 |  |  + Out
 |  |     + LadspaOut
 |  + Plugin
 |  |  + AlsaMidi
 |  |  |  + AlsaMidiIn
 |  |  |  + AlsaMidiOut
 |  |  + AlsaStream
 |  |  |  + AlsaIn
 |  |  |  + AlsaOut
 |  |  + AudioClock
 |  |  |  + AlsaClock
 |  |  |  + Jack
 |  |  + BoundNoise
 |  |  + Chebyshev
 |  |  + Decycler
 |  |  + Delay
 |  |  + DiskStream
 |  |  + FIR
 |  |  + Gain
 |  |  |  + ADSR
 |  |  |  + StereoGain
 |  |  + IIR
 |  |  + IIWU
 |  |  + Inverter
 |  |  + JackIn
 |  |  + JackOut
 |  |  + LFO
 |  |  + LadspaPlugin
 |  |  + Lorenz
 |  |  + MidiIn
 |  |  + MidiOut
 |  |  + MidiPlexer
 |  |  + Modulator
 |  |  + Noise
 |  |  + Part
 |  |  + Pulse
 |  |  + RTClock
 |  |  + RandomSVF
 |  |  + Recycler
 |  |  + Roessler
 |  |  + SamplePlayer
 |  |  + Scheduler
 |  |  |  + DiskScheduler
 |  |  |  + ScriptScheduler
 |  |  + ScriptPlugin
 |  |  + Sine
 |  |  + Spice
 |  |  + SweepingSVF
 |  |  + VCF
 |  |  + VCO
 |  + SoundFile
 |  |  + MpegFile
 |  |  + OggFile
 |  |  + WaveFile
 |  + TimeMap
 + Resampler
 + SampledVoice

ADSR
AlsaClock
AlsaIn
AlsaMidi
AlsaMidiIn
AlsaMidiOut
AlsaOut
AlsaStream
AudioClock
AudioSignal

BoundNoise

Chebyshev
CyclicAudio

DecycledAudio
Decycler
Delay
DiskScheduler
DiskStream

Event
EventCache
EventList

FFT
FIR
FloatEvent

GUSVoice
Gain
Granular
Graph

IIR
IIWU
In
Inverter

Jack
JackIn
JackOut

LFO
LadspaIn
LadspaOut
LadspaPlugin
Lorenz

MidiEvent
MidiIn
MidiOut
MidiPlexer
Modulator
MpegFile

NamedObject
Noise
Note

OSS
Object
ObjectFifo
OggFile
Out

Part
Plugin
Pulse

RTClock
RandomSVF
Recycler
Resampler
Roessler

SamplePlayer
SampledVoice
Scheduler
ScriptPlugin
ScriptScheduler
Sine
SoundFile
Spice
StereoGain
SweepingSVF
SysexEvent

TempoEvent
TextEvent
TimeEvent
TimeMap

VCF
VCO

WaveFile

ADSR ([a = 0.02, d = 0.1, s = 0.7, sr = 24.0, r = 0.1])

AlsaClock ([sample_rate = 44100, frames_per_cycle = 2048, buffer_cycles = 2])

AlsaIn (device[, relaxed = 1])

AlsaMidiIn ('name')

AlsaMidiOut ('name')

AlsaOut (device[, relaxed = 1])

AudioSignal (size)

BoundNoise ([f = .01])

Chebyshev ([h = (0, 1,), c])

Decycler (collect)

Delay ([t = .5, wet = .5])

DiskScheduler()

DiskStream (file)

Event (tick[, loop])

EventList (type)

FFT (size)

FIR ([c = (1,)])

FloatEvent (value[, tick])

Gain ([db = 0.0, n_outs = 1, do_peak = 1])

Granular (audio signal[, gain = 1])

Graph (quarter = 192)

IIR ([ab = ((1,), (0,))])

IIWU (voices[, path = None, flags = 0])

Inverter()

Jack (name)

JackIn (jack, n_ports)

JackOut (jack, n_ports)

LFO ([rate = 144, wave = (0, 1), interpolate = 1])

LadspaPlugin ('ladspa.so'[, name = 'name' or id = -1, subcycle = 0])

Lorenz ([h = .01])

MidiEvent (bytes or attribute = value pairs)

MidiIn ('/path/to/file')

MidiOut ('/path/to/file'[, do_clock = 1])

MidiPlexer()

Modulator ([mul = 1.0, add = 0.0])

MpegFile ('/path/file.mp3'[, stream=0])

NamedObject (name)

Noise ([method = 0, rows = 12])

Note ([note = 'Eb4', v = 100, l = 24, tick = 0])

OSS ([path = '/dev/dsp', channels = 1, sample_rate = 44100, blocksize = 4096])

Object()

ObjectFifo (size[, name])

OggFile ('file.ogg')

Part (list)

Pulse ([period = 2048, width = 1])

RTClock ([frequency = 1024])

RandomSVF ([f, Q])

Recycler()

Resampler (ratio, type = 0)

Lorenz ([h = .01])

SamplePlayer ([channels = 2, db = -12])

SampledVoice (sample[, property = value, ...])

Scheduler()

ScriptPlugin()

ScriptScheduler()

Sine (f)

Spice (gain = 0)

StereoGain ([db = 0, do_peak = 0])

SweepingSVF ([f, Q])

SysexEvent ([data, tick])

TempoEvent ([bpm = 96.0, tick = 0])

TextEvent ([text = ''])

TimeEvent ([n = 4, d = 4, ticks = 24, tick = 0])

TimeMap (list)

VCF ([f, resonance])

VCO ([f = n2f ('c'), shape = 0, pw = 1])

WaveFile ('file.wav'[, mode = 'r', channels = 1, sample_rate = 44100, bits = 0])

RGB

FreeType

Gtk

Glyph

Pixels

PixelView
 + CurveView
 + RGBView
    + DataView
    + MidiView
    + PoleZeroView

CurveView

DataView

FreeType

Glyph
Gtk

MidiView

PixelView
Pixels
PoleZeroView

RGB
RGBView

CurveView (size[, label = ''])

DataView()

FreeType()

Gtk()

MidiView()

PixelView()

Pixels (widget, width, height)

PoleZeroView()

RGB (width, height)

RGBView()