Windows

The primary visual output from a Flitter program is normally through windows. Windows are explicitly created rather than there being some default output space. This allows for multiple windows to be created and controlled simultaneously with a single program. It also allows properties of the windows to be controlled.

In order to render anything, a window rendering tree is defined by placing one or more output nodes inside the window. Each of these rendering nodes creates (at least) one OpenGL texture as an output. Some of these nodes may have children and the textures generated by these become input textures to those nodes.

The top-level window rendering nodes are:

!window

An on-screen window

!offscreen

An off-screen window

These may contain one or more of the following child nodes, each of which creates an output texture.

!image

Loads an image from an external file

!video

Loads and plays a video from an external file

!canvas

A 2D drawing canvas

!canvas3d

A 3D drawing canvas

!record

Composites together its child nodes, records the result to an external file as well as returning this as its output texture

!reference

Allows the output texture of a named window rendering node to be used elsewhere in the tree

!shader

An OpenGL shader program which may have multiple child nodes - see Shaders

A number of built-in shader programs are also available. See Built-in image filters and generators.

Common attributes

All window rendering nodes share a common structure and so they share some common attributes:

size= WIDTH;HEIGHT

Specifies the size of the texture that this node will render into. This value is inherited from the parent node if not specified or, in the case of !image, matched to the content.

id= ID

Specifies a string or symbol identifier for this node that allows the output texture from it to be referenced from elsewhere – either a !reference node, a texture map on a 3D model, or as an image or pattern in a 2D drawing.

hidden= [ true | false ]

This is valid on any child node (i.e., not on !window or !offscreen). Setting this attribute to true will cause the parent node to ignore this node as a child. The output will still be rendered, which means it can still be referenced via an id.

In addition, !window, !offscreen, !video, !record and !shader are all program nodes that run an OpenGL shader program. These programs can be changed with the attributes:

vertex= STRING

Specifies an override vertex shader as a text string containing the GLSL code. Usually this would be read from a file with the read() built-in function. If unspecified, a standard internal shader will be used.

fragment= STRING

Specifies an override fragment shader as a text string containing the GLSL code. Usually this would be read from a file with the read() built-in function. If unspecified, a standard internal shader will be used.

The shader program for !video has a specific function related to the rendering of video frames and so changing this shader is not advised unless you know what you are doing. However, the other nodes follow a standard scheme that is described in the Shaders documentation.

Warning

Although the !canvas3d node supports vertex and fragment shader attributes, these actually override the model instance shader program for the default render group. This is a much more specialised program and writing a new one is a more complicated endeavour.

!window and !offscreen

The !window and !offscreen nodes are largely identical except for the latter not opening on-screen. !offscreen nodes are primarily intended to collect window rendering nodes that are to be used as references rather than through direct rendering into a window. However, all !window nodes can be made to behave as !offscreen nodes with the --offscreen command-line option. This can be useful for saving output to a file (with a !record node) without opening a window.

!window and !offscreen nodes support the following specific attribute:

colorbits= [ 8 | 16 | 32 ]

This specifies the default bit depth of output texture color channels for the window rendering tree. If not specified, it defaults to 16 bits. The color depth of the actual !window on-screen frame-buffer cannot be controlled and is OS-defined.

The !window node also supports the following specific attributes:

screen= 0…

Specifies which screen to open the window on. This is OS and configuration dependent, but generally screen 0 is the “main” screen and additional screens are numbered upwards from that. Default is 0 unless the --screen command-line option has been provided.

fullscreen= [true | false]

Specifies whether to show the window in full-screen mode. If set to true, the window will be expanded to fill the entire screen. If the window size does not match the aspect ratio of the screen then black borders will be shown at the top/bottom or left/right sides as necessary. This attribute may be changed programmatically to switch the window in and out of full-screen mode. Default is false unless the --fullscreen command-line option has been provided.

cursor= [true | false]

Whether the pointer cursor should be shown within the window. This attribute may be changed programmatically to show and hide the cursor. Default is true unless the window is in full-screen mode.

The default shader program used for !window and !offscreen nodes is a single-pass shader that composites together the output textures of all child nodes. It can be controlled with the following additional attributes:

composite= [ :over | :dest_over | :lighten | :darken | :add | :difference | :multiply ]

Specifies the blend function to use, default :over.

alpha= ALPHA

Specifies a final alpha value to be applied to the entire shader output, default 1.

Note

By default, !shader nodes run the same simple compositing shader program as !window. Therefore a “bare” !shader node (one without a custom program specified with the fragment and/or vertex attributes) can be used to composite together child nodes. This is most useful where a blend function different to the default :over function is required.

!key and !pointer

!window nodes support a basic input system similar to controllers that allows keyboard and pointer input to be connected to the state system. This is controlled by adding one or more !key nodes as children of the !window node and/or a !pointer node.

!key nodes support the following attributes:

state= PREFIX

The prefix for state keys related to this key.

name= NAME

The name of the key as a string or symbol. The key names are those defined by GLFW (in lowercase and without the leading GLFW_KEY_ prefix).

A !key node must be given for each key that the program is interested in. The following entries will be created in the state mapping for each key:

PREFIX

A value of true if the key is currently pressed, false if it is released or null if this is unknown.

PREFIX ;pushed

This is the same value as the PREFIX key.

PREFIX ;pushed;:beat

The beat counter value at the moment that the key was last pressed, or null if this event has not yet occurred.

PREFIX ;released

This is the logical negation of the the PREFIX key value, i.e., true if the key is currently released, false if it is pressed or null if this is unknown.

PREFIX ;released;:beat

The beat counter value at the moment that the key was last released, or null if this event has not yet occurred.

A !pointer node supports just the state attribute and creates the following entries in the state mapping:

PREFIX

The current pointer position as a 2-item vector normalized to the \([0,1]\) range, where \(0\) is the left/top of the window and \(1\) is the right/bottom. If the pointer is not within the bounds of the window then this state key will be null.

PREFIX ; (0 | 1 | … )

The status of the pointer button(s), numbered from 0 upwards – which of these is “left” or “right” is OS dependent. The state value will be true if the pointer button is currently pressed, false if it is released or null if the state is not currently known (for instance the window has just opened and no pointer events have been processed).

!image

An !image node loads the contents of an external image file into a texture for use in the window rendering tree. This might be as an input to a !shader, for displaying static slideshow images in a !window, or in an !offscreen to use as a referenced texture for 3D model texture mapping.

There are two supported attributes:

filename= PATH

Specifies the path of the file to load with respect to the location of the running Flitter program.

size= WIDTH;HEIGHT

If specified, then the image will be resized to this size with bilinear interpolation.

Unlike the rest of the window rendering nodes, !image does not inherit its size from its parent. If size is not specified, then the output texture size will match the pixel dimensions of the loaded image.

The !image output texture is only changed if the underlying file changes, or the filename or size attributes are changed. This makes it a very cheap node to render compared to, say, creating a !canvas node and drawing an image into it. !image can load all image file types supported by the Pillow image library.

!video

A !video node loads and renders frames from an external video file into a texture. The texture will inherit the size attribute as normal, or this can be specified explicitly. By default, the video will be stretched to fill this size, but this can be controlled with the aspect attribute described below.

The following attributes are supported:

filename= PATH

Specifies the path of the video to open with respect to the location of the running Flitter program.

position= TIMESTAMP

Specifies the time-stamp, in seconds from the start of the video, of the frame to be output.

loop= [ true | false ]

Specifies whether the time-stamps beyond the end of the video will loop around to the beginning. This also enables negative time-stamps, which will loop around to the end. If set to false (the default), time-stamps outside of the video range will be clamped to the first or last frames of the video.

back_and_forth= [ true | false ]

If loop=true and back_and_forth=true then timestamps beyond the end of the video will work backwards through the video to the start and then forwards again. This is useful for textural videos to avoid a discontinuity but may cause performance problems depending on the video encoding (see note below).

trim= START;END

Specifies an amount of time in seconds to trim off the beginning and end of the video. This affects how position is mapped to the source video and the operation of loop.

interpolate= [ true | false ]

Specifies whether to mix two successive frames if position references a value between the frame time-stamps. This can be useful for generating slow-motion output if the video does not contain much movement. The default is false.

aspect= [ :fit | :fill ]

If the source video is a different aspect ratio to that of the node size, then this specifies that the video aspect ratio should be respected and the video either scaled to fit in the frame (with borders on the top/bottom or left/right sides) or scaled to fill the entire frame (with the video cropped at the top/bottom or left/right sides). If not specified, then the default is to stretch the video to fill the frame.

thread= [ true | false ]

Specifies whether to use multi-threaded video decoding. This has higher overall performance, but introduces a small delay on the first frame. This delay may be unacceptable if the video needs to start immediately on load.

gamma= ALPHA

Specifies a gamma curve correction to be applied to the video frames, default 1 (i.e., no correction). Values less than 1 will lighten the output image and values greater than 1 will darken it.

alpha= ALPHA

Specifies a final alpha value to be applied to the video frames, default 1.

A video is played by setting the position attribute to a time-varying value in code such as the time global name. There is no requirement for this value to vary in real-time, it can slow to a stop or run faster. position may skip forward or backwards by a large step, which will cause a frame seek to the new location.

Warning

position may run backwards. However there is an important caveat: if the video makes extensive use of P-frames then this will cause a slight judder at each I-frame boundary. The video player will need to seek back to the previous I-frame and then decode forwards to the desired frame. The in-between frames are cached, but the same seek and decode forwards will have to be done each time an I-frame is hit.

!video uses the PyAV library, which is a wrapper around ffmpeg. It thus supports a wide range of video file types (including animated GIF files).

!record

The !record node expects one or more child nodes, which will be composited together as necessary and written to an image or video file. Whether to write an image or a video is (in general) selected automatically based on the filename extension (e.g., .jpg will write a JPEG image file).

!record supports the following attributes:

filename= PATH

Specifies the path of the image or video file to write to, with respect to the location of the running Flitter program. If filename is null, then the !record node will do nothing – this is a simple way to delay output until a particular condition holds.

keep_alpha= BOOLEAN

Specifies whether to keep the alpha transparency layer information from the input when writing the image or video file. Only some image formats (such as PNG or JPEG2000) and a very few video codecs (such as ProRes) support transparency. Default is false.

quality= Q

Specifies a quality setting for image formats that support it (such as JPEG).

codec= CODEC

For video writing, this specifies the video codec to use. Defaults to :h264.

pixfmt= PIXFMT

For video writing, this specifies the video frame pixel format to be passed to the encoder. For most codecs the default yuv420p – or yuva420p with keep_alpha=true – will be correct.

limit= SECONDS

Specifies a maximum number of seconds of video output to write before closing the file. Otherwise, the video output will continue for as long as filename is valid and the program is running.

For video writing, any further attributes specified on the !record node will be passed to ffmpeg as options. For ffmpeg options that use dashes (-), replace each of these with a sequential pair of underscores (__).

Some common, useful codec options are:

crf= CRF

For :h264 and :hevc, this provides a “constant rate factor” that defines how much the codec should prioritise size over quality. Smaller values mean better quality and larger values mean a smaller size. A value around 25 is generally an acceptable compromise for the :h264 codec. For :hevc, this can often be pushed up to a higher value for smaller files while still keeping a decent quality encoding.

profile= PROFILE

For many video codecs this controls a set of features of the codec that have been standardised to the capabilities of different playback devices. For instance, a profile may be designed for a high-end broadcast context or mobile phones. Common profiles have names like :baseline, :main or :high.

preset= PRESET

For many video codecs this groups up different codec settings by simpler names. Common presets have names like :fast or :slow.

The full range of per-codec options can be found in the ffmpeg codec documentation or by running ffmpeg -h codec=CODEC.

Filenames with an .mp4, .mov, .m4v, .mkv, .webm or .ogg extension are assumed to be video outputs with the appropriate container type. For video output, the specific codec should be selected with the codec attribute described below.

If the extension is .gif then a static GIF image file will be written by default. However if codec=:gif is also supplied, then video output is assumed and an animated GIF file will be written.

The output that will be written to the file is also passed through as the output of the node, therefore a !record node can be inserted at any point in the window hierarchy. Multiple !record nodes can be used in the same hierarchy to record different outputs at the same time and !record nodes may be nested to simultaneously record a raw and processed output (with the important caveat that the output of a !record node is always 8-bit sRGB).

A particular image file will be written once per run of a Flitter program, i.e., once an image has been written to a particular file, the !record node will do nothing. However, the filename attribute can be changed to record a new image. In this way, a constantly changing filename can be used to write individual animation frames as images. For example, this program will write a new JPEG snapshot into the output folder on every frame:

!window …
    !record filename='output/frame';frame;'.jpg' quality=90
        …

Note

If the Flitter engine is running in realtime mode, !record will attempt to record a “live” video. The frame rate of the output video will be variable and will depend on how fast the encoder can run.

The video encoding runs on a background thread with a 1-second frame buffer. On a fast computer, this can be used to record a decent video of a live performance. However, if the encoder is unable to keep up with the running frame rate of the engine, then live frames will be dropped and the output video will contain stutters.

To record clean videos with a fixed frame rate, the engine should be run in non-realtime mode with the --lockstep command-line option.

!reference

The output texture of one node in a window rendering tree can be used in multiple places in the tree with a !reference node. The node takes a single attribute:

id= ID

Specifies the value of the id attribute of matching node.

For example, a bloom-filter pipeline might look like this:

let SIZE=1920;1080

!window size=SIZE composite=:lighten
    !canvas3d id=:bloom_source
        …
    !shader fragment=read('blur.frag') radius=5
        !shader fragment=read('threshold.frag') level=0.5
            !reference id=:bloom_source

This allows the !canvas3d output to appear at two places in the window rendering tree: as a direct child of the window, and as an input to the threshold.frag shader program.

References within a single !window tree are updated in render order: the children of a node are rendered in-order before rendering the node. So a !reference node that refers to a node that has already been rendered will use the current rendered image of that node. A reference to a node that occurs later (including any parent node of the !reference) is valid only after the first frame and will return the rendered image from the previous frame.

A !reference to a node outside of the enclosing !window (or !offscreen) is also valid only after the first frame and will return the rendered texture from either the current or previous frame. No guarantees are made about the render order of top-level nodes.