# Windows The primary visual output from a **Flitter** program is normally through windows. Windows are explicitly created rather than there being some default output space. This allows for multiple windows to be created and controlled simultaneously with a single program. It also allows properties of the windows to be controlled. In order to render anything, a *window rendering tree* is defined by placing one or more output nodes inside the window. Each of these rendering nodes creates (at least) one OpenGL texture as an output. Some of these nodes may have children and the textures generated by these become input textures to those nodes. The top-level window rendering nodes are: `!window` : An on-screen window `!offscreen` : An off-screen window These may contain one or more of the following child nodes, each of which creates an output texture. `!image` : Loads an image from an external file `!video` : Loads and plays a video from an external file `!canvas` : A [2D drawing canvas](canvas.md) `!canvas3d` : A [3D drawing canvas](canvas3d.md) `!record` : Composites together its child nodes, records the result to an external file as well as returning this as its output texture `!reference` : Allows the output texture of a named window rendering node to be used elsewhere in the tree `!shader` : An OpenGL shader program which may have multiple child nodes - see [Shaders](shaders.md) A number of built-in shader programs are also available. See [Built-in image filters and generators](shaders.md#built-in-image-filters-and-generators). ## Common attributes All window rendering nodes share a common structure and so they share some common attributes: `size=` *WIDTH*`;`*HEIGHT* : Specifies the size of the texture that this node will render into. This value is inherited from the parent node if not specified or, in the case of `!image`, matched to the content. `id=` *ID* : Specifies a string or symbol identifier for this node that allows the output texture from it to be referenced from elsewhere – either a `!reference` node, a [texture map](canvas3d,md#texture-mapping) on a 3D model, or as an [image](canvas.md#images) or [pattern](canvas.md#pattern) in a 2D drawing. `hidden=` [ `true` | `false` ] : This is valid on any child node (i.e., not on `!window` or `!offscreen`). Setting this attribute to `true` will cause the parent node to ignore this node as a child. The output will still be rendered, which means it can still be referenced via an `id`. In addition, `!window`, `!offscreen`, `!video`, `!record` and `!shader` are all *program nodes* that run an OpenGL shader program. These programs can be changed with the attributes: `vertex=` *STRING* : Specifies an override vertex shader as a text string containing the GLSL code. Usually this would be read from a file with the [`read()` built-in function](builtins.md#file-functions). If unspecified, a standard internal shader will be used. `fragment=` *STRING* : Specifies an override fragment shader as a text string containing the GLSL code. Usually this would be read from a file with the [`read()` built-in function](builtins.md#file-functions). If unspecified, a standard internal shader will be used. The shader program for `!video` has a specific function related to the rendering of video frames and so changing this shader is not advised unless you know what you are doing. However, the other nodes follow a standard scheme that is described in the [Shaders](shaders.md) documentation. :::{warning} Although the `!canvas3d` node supports `vertex` and `fragment` shader attributes, these actually override the model instance shader program for the default [render group](canvas3d.md#render-groups). This is a much more specialised program and writing a new one is a more complicated endeavour. ::: ## `!window` and `!offscreen` The `!window` and `!offscreen` nodes are largely identical except for the latter not opening on-screen. `!offscreen` nodes are primarily intended to collect window rendering nodes that are to be used as references rather than through direct rendering into a window. However, all `!window` nodes can be made to behave as `!offscreen` nodes with the `--offscreen` [command-line option](install,md#running-flitter). This can be useful for saving output to a file (with a [`!record`](#record) node) without opening a window. `!window` and `!offscreen` nodes support the following specific attribute: `colorbits=` [ `8` | `16` | `32` ] : This specifies the default bit depth of output texture color channels for the window rendering tree. If not specified, it defaults to `16` bits. The color depth of the actual `!window` on-screen frame-buffer cannot be controlled and is OS-defined. The `!window` node also supports the following specific attributes: `screen=` `0`... : Specifies which screen to open the window on. This is OS and configuration dependent, but generally screen `0` is the "main" screen and additional screens are numbered upwards from that. Default is `0` unless the `--screen` [command-line option](install.md#running-flitter) has been provided. `fullscreen=` [`true` | `false`] : Specifies whether to show the window in full-screen mode. If set to `true`, the window will be expanded to fill the entire screen. If the window `size` does not match the aspect ratio of the screen then black borders will be shown at the top/bottom or left/right sides as necessary. This attribute may be changed programmatically to switch the window in and out of full-screen mode. Default is `false` unless the `--fullscreen` [command-line option](install.md#running-flitter) has been provided. `cursor=` [`true` | `false`] : Whether the pointer cursor should be shown within the window. This attribute may be changed programmatically to show and hide the cursor. Default is `true` unless the window is in full-screen mode. The default shader program used for `!window` and `!offscreen` nodes is a single-pass shader that composites together the output textures of all child nodes. It can be controlled with the following additional attributes: `composite=` [ `:over` | `:dest_over` | `:lighten` | `:darken` | `:add` | `:difference` | `:multiply` ] : Specifies the blend function to use, default `:over`. `alpha=` *ALPHA* : Specifies a final alpha value to be applied to the entire shader output, default `1`. :::{note} By default, `!shader` nodes run the same simple compositing shader program as `!window`. Therefore a "bare" `!shader` node (one without a custom program specified with the `fragment` and/or `vertex` attributes) can be used to composite together child nodes. This is most useful where a blend function different to the default `:over` function is required. ::: ### `!key` and `!pointer` `!window` nodes support a basic input system similar to [controllers](controllers.md) that allows keyboard and pointer input to be connected to the [state system](language.md#state). This is controlled by adding one or more `!key` nodes as children of the `!window` node and/or a `!pointer` node. `!key` nodes support the following attributes: `state=` *PREFIX* : The prefix for state keys related to this key. `name=` *NAME* : The *name* of the key as a string or symbol. The key names are those defined by [GLFW](https://www.glfw.org/docs/latest/group__keys.html) (in lowercase and without the leading `GLFW_KEY_` prefix). A `!key` node must be given for each key that the program is interested in. The following entries will be created in the state mapping for each key: *PREFIX* : A value of `true` if the key is currently pressed, `false` if it is released or `null` if this is unknown. *PREFIX* `;pushed` : This is the same value as the *PREFIX* key. *PREFIX* `;pushed;:beat` : The beat counter value at the moment that the key was *last* pressed, or `null` if this event has not yet occurred. *PREFIX* `;released` : This is the logical negation of the the *PREFIX* key value, i.e., `true` if the key is currently *released*, `false` if it is *pressed* or `null` if this is unknown. *PREFIX* `;released;:beat` : The beat counter value at the moment that the key was *last* released, or `null` if this event has not yet occurred. A `!pointer` node supports just the `state` attribute and creates the following entries in the state mapping: *PREFIX* : The current pointer position as a 2-item vector normalized to the $[0,1]$ range, where $0$ is the left/top of the window and $1$ is the right/bottom. If the pointer is not within the bounds of the window then this state key will be `null`. *PREFIX* `;` (`0` | `1` | … ) : The status of the pointer button(s), numbered from `0` upwards – which of these is "left" or "right" is OS dependent. The state value will be `true` if the pointer button is currently pressed, `false` if it is released or `null` if the state is not currently known (for instance the window has just opened and no pointer events have been processed). ## `!image` An `!image` node loads the contents of an external image file into a texture for use in the window rendering tree. This might be as an input to a `!shader`, for displaying static slideshow images in a `!window`, or in an `!offscreen` to use as a referenced texture for [3D model texture mapping](canvas3d.md#texture-mapping). There are two supported attributes: `filename=` *PATH* : Specifies the path of the file to load with respect to the location of the running **Flitter** program. `size=` *WIDTH*`;`*HEIGHT* : If specified, then the image will be resized to this size with bilinear interpolation. `flip=` [`:horizontal` | `:vertical` | `:both`] : If specified, then the image will be flipped in this direction, i.e., left-to-right for `:horizontal` and top-to-bottom for `:vertical`. Unlike the rest of the window rendering nodes, `!image` does not inherit its size from its parent. If `size` is not specified, then the output texture size will match the pixel dimensions of the loaded image. The `!image` output texture is only changed if the underlying file changes, or the `filename` or `size` attributes are changed. This makes it a very cheap node to render compared to, say, creating a `!canvas` node and drawing an image into it. `!image` can load all [image file types supported by the **Pillow** image library](https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html). ## `!video` A `!video` node loads and renders frames from an external video file into a texture. The texture will inherit the `size` attribute as normal, or this can be specified explicitly. By default, the video will be stretched to fill this size, but this can be controlled with the `aspect` attribute described below. The following attributes are supported: `filename=` *PATH* : Specifies the path of the video to open with respect to the location of the running **Flitter** program. `position=` *TIMESTAMP* : Specifies the time-stamp, in seconds from the start of the video, of the frame to be output. `loop=` [ `true` | `false` ] : Specifies whether the time-stamps beyond the end of the video will loop around to the beginning. This also enables negative time-stamps, which will loop around to the end. If set to `false` (the default), time-stamps outside of the video range will be clamped to the first or last frames of the video. `back_and_forth=` [ `true` | `false` ] : If `loop=true` and `back_and_forth=true` then timestamps beyond the end of the video will work backwards through the video to the start and then forwards again. This is useful for textural videos to avoid a discontinuity but may cause performance problems depending on the video encoding (see note below). `trim=` *START*`;`*END* : Specifies an amount of time in seconds to trim off the beginning and end of the video. This affects how `position` is mapped to the source video and the operation of `loop`. `interpolate=` [ `true` | `false` ] : Specifies whether to mix two successive frames if `position` references a value between the frame time-stamps. This can be useful for generating slow-motion output if the video does not contain much movement. The default is `false`. `aspect=` [ `:fit` | `:fill` ] : If the source video is a different aspect ratio to that of the node `size`, then this specifies that the video aspect ratio should be respected and the video either scaled to *fit* in the frame (with borders on the top/bottom or left/right sides) or scaled to *fill* the entire frame (with the video cropped at the top/bottom or left/right sides). If not specified, then the default is to stretch the video to fill the frame. `thread=` [ `true` | `false` ] : Specifies whether to use multi-threaded video decoding. This has higher overall performance, but introduces a small delay on the first frame. This delay may be unacceptable if the video needs to start immediately on load. `gamma=` *ALPHA* : Specifies a gamma curve correction to be applied to the video frames, default `1` (i.e., no correction). Values less than 1 will lighten the output image and values greater than 1 will darken it. `alpha=` *ALPHA* : Specifies a final alpha value to be applied to the video frames, default `1`. A video is played by setting the `position` attribute to a time-varying value in code such as the `time` global name. There is no requirement for this value to vary in real-time, it can slow to a stop or run faster. `position` may skip forward or backwards by a large step, which will cause a frame seek to the new location. :::{warning} `position` may run *backwards*. However there is an important caveat: if the video makes extensive use of [P-frames](https://en.wikipedia.org/wiki/Video_compression_picture_types) then this will cause a slight judder at each I-frame boundary. The video player will need to seek back to the previous I-frame and then decode forwards to the desired frame. The in-between frames are cached, but the same seek and decode forwards will have to be done each time an I-frame is hit. ::: `!video` uses the [**PyAV** library](https://pyav.org), which is a wrapper around [**ffmpeg**](https://ffmpeg.org). It thus supports a wide range of video file types (including animated GIF files). ## `!record` The `!record` node expects one or more child nodes, which will be composited together as necessary and written to an image or video file. Whether to write an image or a video is (in general) selected automatically based on the filename extension (e.g., `.jpg` will write a JPEG image file). `!record` supports the following attributes: `filename=` *PATH* : Specifies the path of the image or video file to write to, with respect to the location of the running **Flitter** program. If `filename` is `null`, then the `!record` node will do nothing – this is a simple way to delay output until a particular condition holds. `keep_alpha=` *BOOLEAN* : Specifies whether to keep the alpha transparency layer information from the input when writing the image or video file. Only some image formats (such as PNG or JPEG2000) and a very few video codecs (such as ProRes) support transparency. Default is `false`. `quality=` *Q* : Specifies a quality setting for image formats that support it (such as JPEG). `codec=` *CODEC* : For video writing, this specifies the video codec to use. Defaults to `:h264`. `pixfmt=` *PIXFMT* : For video writing, this specifies the video frame pixel format to be passed to the encoder. For most codecs the default `yuv420p` – or `yuva420p` with `keep_alpha=true` – will be correct. `limit=` *SECONDS* : Specifies a maximum number of seconds of video output to write before closing the file. Otherwise, the video output will continue for as long as `filename` is valid and the program is running. For video writing, any further attributes specified on the `!record` node will be passed to **ffmpeg** as options. For **ffmpeg** options that use dashes (`-`), replace each of these with a sequential pair of underscores (`__`). Some common, useful codec options are: `crf=` *CRF* : For `:h264` and `:hevc`, this provides a "constant rate factor" that defines how much the codec should prioritise size over quality. Smaller values mean better quality and larger values mean a smaller size. A value around `25` is generally an acceptable compromise for the `:h264` codec. For `:hevc`, this can often be pushed up to a higher value for smaller files while still keeping a decent quality encoding. `profile=` *PROFILE* : For many video codecs this controls a set of features of the codec that have been standardised to the capabilities of different playback devices. For instance, a profile may be designed for a high-end broadcast context or mobile phones. Common profiles have names like `:baseline`, `:main` or `:high`. `preset=` *PRESET* : For many video codecs this groups up different codec settings by simpler names. Common presets have names like `:fast` or `:slow`. The full range of per-codec options can be found in the [**ffmpeg** codec documentation](https://ffmpeg.org/ffmpeg-codecs.html) or by running `ffmpeg -h codec=`*CODEC*. Filenames with an `.mp4`, `.mov`, `.m4v`, `.mkv`, `.webm` or `.ogg` extension are assumed to be video outputs with the appropriate container type. For video output, the specific codec should be selected with the `codec` attribute described below. If the extension is `.gif` then a static GIF image file will be written by default. However if `codec=:gif` is *also* supplied, then video output is assumed and an animated GIF file will be written. The output that will be written to the file is also passed through as the output of the node, therefore a `!record` node can be inserted at any point in the window hierarchy. Multiple `!record` nodes can be used in the same hierarchy to record different outputs at the same time and `!record` nodes may be nested to simultaneously record a raw and processed output (with the important caveat that the output of a `!record` node is always 8-bit sRGB). A particular image file will be written once per run of a **Flitter** program, i.e., once an image has been written to a particular file, the `!record` node will do nothing. However, the `filename` attribute can be changed to record a new image. In this way, a constantly changing filename can be used to write individual animation frames as images. For example, this program will write a new JPEG snapshot into the `output` folder on every frame: ```flitter !window … !record filename='output/frame';frame;'.jpg' quality=90 … ``` :::{note} If the **Flitter** engine is running in realtime mode, `!record` will attempt to record a "live" video. The frame rate of the output video will be variable and will depend on how fast the encoder can run. The video encoding runs on a background thread with a 1-second frame buffer. On a fast computer, this can be used to record a decent video of a live performance. However, if the encoder is unable to keep up with the running frame rate of the engine, then live frames will be dropped and the output video will contain stutters. To record clean videos with a fixed frame rate, the engine should be run in non-realtime mode with the `--lockstep` command-line option. ::: ## `!reference` The output texture of one node in a window rendering tree can be used in multiple places in the tree with a `!reference` node. The node takes a single attribute: `id=` *ID* : Specifies the value of the `id` attribute of matching node. For example, a bloom-filter pipeline might look like this: ```flitter let SIZE=1920;1080 !window size=SIZE composite=:lighten !canvas3d id=:bloom_source … !shader fragment=read('blur.frag') radius=5 !shader fragment=read('threshold.frag') level=0.5 !reference id=:bloom_source ``` This allows the `!canvas3d` output to appear at two places in the window rendering tree: as a direct child of the window, and as an input to the `threshold.frag` shader program. References *within* a single `!window` tree are updated in render order: the children of a node are rendered in-order before rendering the node. So a `!reference` node that refers to a node that has already been rendered will use the *current* rendered image of that node. A reference to a node that occurs later (including any parent node of the `!reference`) is valid only *after* the first frame and will return the rendered image from the previous frame. A `!reference` to a node *outside* of the enclosing `!window` (or `!offscreen`) is also valid only after the first frame and will return the rendered texture from *either the current or previous frame*. No guarantees are made about the render order of top-level nodes.