Spatial Audio Programming for Technological Arts

Position

The Society for Arts and Technology [SAT] needs you for an immediate start at Metalab, the SAT research laboratory. Your role will be to participate in the research process: technology watch, ideation, development, implementation, communication and participation in partnership projects. You will participate in the evolution, code review and use of Metalab software, in particular for our audio spatialization engine (SATIE) and our acoustic simulation tool (vaRays). An experience of at least 5 years is desired.

Technical environment

  • Ubuntu, Linux, FOSS

  • SuperCollider, Numpy

  • C++, Python3

  • Gitlab, JIRA, Confluence, Slack

Assets

  • Spatial audio, ambisonics

  • Acoustics

  • Blender

  • GDB, cmake, clang

Work environment

Since 2002, the Metalab is the Society for Arts and Technology [SAT]’s research and development lab. The Metalab has a dual mission: to stimulate the emergence of new immersive experiences, and to make their design and authoring accessible to artists and creators through an ecosystem of free software.

The Metalabs research fields – telepresence, immersion, video mapping, and spatialized sound – are explored through research and production projects. The Metalab team is comprised of 8 to 15 people with various technical expertise including digital audio, computer graphics, networks, software development and multimedia integration. The Metalab welcomes interns in a variety of subjects, even outside the strict scope of the laboratory expertises.

The Metalab's works aims to cover the whole pipeline of immersive and distributed content production. The Metalab’s software addresses, among others, immersive in situ editing, video mapping, audio spatialization, group interaction and low latency transmission. They are designed as flexible tools, and supports Metalab’s research and partnerships. This software suite is used by the SAT during creative residencies and is at the heart of telepresence projects, between libraries (Bibliolab) and between venues (Scènes ouvertes). Metalab’s partners include artists, university researchers, schools, designers and private companies.

Metalab Software

  • Immersive Space Editing and Prototyping: In situ editing (EiS)

  • Multi-projector video mapping for all types of surfaces: Splash

  • Calibration of projectors for immersive space: Calimiro

  • Sound spatialization for all types of speaker systems: SATIE

  • Sound spatialization by acoustic simulation: vaRays

  • Group interaction of people in an immersive space: LivePose

  • Telepresence and multichannel transmission in low latency: Switcher

  • Sharing of all types of data streams between applications: Shmdata

Development of an open-source toolkit for creating haptic effects with ultrasound displays

Internship title

Development of an open-source toolkit for creating haptic effects with ultrasound displays

Context

Founded in 2002, Metalab is the research laboratory of the Society for Arts and Technology [SAT]. Metalab's mission is twofold:

  1. to stimulate the emergence of innovative immersive experiences,
  2. to make their design accessible to artists and creators of immersion through an ecosystem of free software.

Besides our set of tools for audio- and visual rendering, we are exploring ways to enhance immersive experiences with haptic feedback, to create simulations that stimulate the sense of touch. One first haptic rendering device created at Metalab is the Scalable Haptic Floor that transmits haptic feedback to our bodies through their points of contact with the floor. In addition, ultrasound haptics render air-borne haptic feedback in mid-air, without contact, and without requiring to wear devices [1,2], opening new perspectives for immersion through focused haptic feedback.

Objective

The main objective of this internship is to develop an open-source toolkit for creating haptic feedback with ultrasound displays, reusing recent research works [1,2], and facilitating the authoring of ultrasound haptic effects for artists and creators of immersion.

This toolkit could complement other Metalab tools, for instance using Poire for haptic generation through audio signals as proxy, similarly to the Scalable Haptic Floor, and LivePose for posture or hand detection to support interactive scenarios.

Tasks

  • Review haptic effects that can be generated with ultrasound displays.
  • Develop the open-source toolkit (hardware and software building upon recent research works [1,2]).
  • Create a demo video.
  • Participate in the life of the laboratory: scrums, code review, etc.
  • Document our work in a lab notebook and git repositories and ensure its reproducibility.
  • Co-author a publication in the scope of this internship.

Work environment

  • Hardware deployment: Arduino or Raspberry Pi
  • Software development: C++, JavaScript or Python
  • Documentation: BibTeX, LaTeX, Markdown
  • Workflows: Confluence, Gitlab, Jira
  • Ecosystem: Linux, Free and Open Source Software (FOSS)

References

Development of an open-source toolkit for creating deformable haptic surfaces

Internship title

Development of an open-source toolkit for creating deformable haptic surfaces

Context

Founded in 2002, Metalab is the research laboratory of the Society for Arts and Technology [SAT]. Metalab's mission is twofold:

  1. to stimulate the emergence of innovative immersive experiences,
  2. to make their design accessible to artists and creators of immersion through an ecosystem of free software.

Besides our set of tools for audio- and visual rendering, we are exploring ways to enhance immersive experiences with haptic feedback, to create simulations that stimulate the sense of touch. One first haptic rendering device created at Metalab is the Scalable Haptic Floor that transmits haptic feedback to our bodies through their points of contact with the floor. In addition, deformable haptic surfaces render haptic feedback through peripheral surfaces above the ground [1], opening new perspectives for immersion through interactive haptic feedback beyond desk-based settings [2].

Objective

The main objective of this internship is to develop an open-source toolkit for creating haptic feedback on deformable surfaces, reusing recent research works [1], and facilitating the authoring of deformable haptic interactions for artists and creators of immersion.

This toolkit could complement other Metalab tools, for instance informing Splash of surface deformations for video mapping on deformable surfaces.

Tasks

  • Review related actuation techniques.
  • Develop the open-source toolkit (hardware and software).
  • Create a demo video.
  • Participate in the life of the laboratory: scrums, code review, etc.
  • Document our work in a lab notebook and git repositories and ensure its reproducibility.
  • Co-author a publication in the scope of this internship.

Work environment

  • Hardware deployment: Arduino or Raspberry Pi
  • Software development: C++, JavaScript or Python
  • Documentation: BibTeX, LaTeX, Markdown
  • Workflows: Confluence, Gitlab, Jira
  • Ecosystem: Linux, Free and Open Source Software (FOSS)

References

Technical writing

Internship title

Technical writing

Goal

You like learning new technologies? Moreso, you like explaining them to others? Searching for the right word does not scare you? We would like to meet you!

The goal of the technical writing internship is to play an active role in creating and writing new tutorials for at least one of the following software projects: Splash, shmdata and LivePose.

The Metalab team write, edit and maintain documentation websites to facilitate the discovery and use of their tools.

Tasks

With the support of the Metalab team:

  • Explore the Metalab tools by using them with their documentation, with the goal of understanding their main usecases.
  • Identify usecases to be documented, with the help of a Metalab team member.
  • Make a plan for the new tutorials to be written.
  • Write the new tutorials.
  • Improve the tutorials according to comments by the team.

Work environment

For the documentation, our main tools are Sphinx (markdown or rst format) and GitLab.

Our environment promotes team work and the adoption of open source software, including:

  • Scrums and code reviews
  • GitLab, Markdown
  • Linux, Free and Open Source Software (FOSS)
  • LaTeX

Audiodice speaker calibration

Internship title

Audiodice calibration

Objective

This interships aims to calibrate audiodice speakers, a set of 5 dodecahedron with one speaker per side.

https://vimeo.com/519938720

This type of speaker needs to be calibrated in two ways: - Frequency response - Directivity of the speaker depending it's configuration (single speaker at once, omnidirectional pattern)

From this calibration data, we will be able to create a profile of the audiodice and use it to enhance the sound reproduction. It will also help to create audio spatializer thanks to the directivity knowledge.

Tasks

  • Participate in the measurements
  • Explore existing in-house and other tools for similar tasks
  • Experiment and develop or improve existing in-house tools or pipelines involving other tools
  • Participate in the development of a demo (video or live)
  • Participate in the life of the laboratory: scrums, code review, etc.
  • Document the work and ensure its reproducibility.

Context and software

  • Linux OS
  • Digital signal processing
  • Python (numpy), bash
  • 3D audio and ambisonics
  • Optional: SuperCollider

SATIE - toolbox integrations

Internship title

SATIE - toolbox integrations

Objective

SATIE is our in-house audio spatializer that aims to be flexible, robust and supports many different arrangements of speaker arrays. We are pondering a toolbox/middleware in an effort to make it more approachable and useful to musicians, composers and sound artists who are not programmers. The objective of this internship is to help us to identify possible tools and techniques to make SATIE more easily integrable to typical spatial audio projects. The ideal candidate will have experience and understanding with interactive music digital pipelines and be able and interested to tackle technical aspects of integrating software pipelines.

Tasks

  • Participate in the ideation
  • Develop prototypes of integration pipelines involving music production tools and SATIE
  • Participate in the development of a demo (video or live)
  • Participate in the life of the laboratory: scrums, code review, etc.
  • Document the work and ensure its reproducibility.

Context and software

  • Mostly Linux OS
  • Digital signal processing
  • SuperCollider, Bash, Python
  • 3D audio, show control
  • Possible building blocks: Max4Live, OSSIAscore, Chataigne,

Computer Graphics and Computer Vision Programming for the Technological Arts (closed)

Position

The Society for Arts and Technology [SAT] is looking for a developer-researcher in computer graphics and computer vision to start immediately in its research laboratory. Your role will be to participate in the research process: monitoring, ideation, development, implementation, communication and partnership projects. In particular, you will participate in the work surrounding our video mapper (Splash), our structured light photogrammetry library (Calimiro) and our multi-camera pose estimation tool (Spook).

Technical environment

  • Ubuntu, Linux, FOSS

  • OpenGL, OpenCV, Tensorflow, PyTorch, Numpy

  • C++, CMake, Python

  • Doxygen, Sphinx

  • NVIDIA Jetson

  • Gitlab, JIRA, Confluence, Slack

Assets

  • Real-time 3D programming

  • Image processing

  • Machine learning

  • Interactive installations

  • Implementation and redaction of scientific articles

Work environment

Founded in 2002, the Metalab is the Society for Arts and Technology [SAT]’s research and development lab. The Metalab has a dual mission: to stimulate the emergence of new immersive experiences, and to make their design and authoring accessible to artists and creators through an ecosystem of free software.

The Metalab's research fields – telepresence, immersion, video mapping, and spatialized sound – are explored through research and production projects. The Metalab team is comprised of 8 to 15 people with various technical expertise including digital audio, computer graphics, networks, software development and multimedia integration. The Metalab welcomes interns in a variety of subjects, even outside the strict scope of the laboratory expertise.

The Metalab's work aims to cover the whole pipeline of immersive and distributed content production. The Metalab’s software addresses, among others, immersive in situ editing, video mapping, audio spatialization, group interaction and low latency transmission. They are designed as flexible tools, and support Metalab’s research and partnerships. This software suite is used by the SAT during creative residencies and is at the heart of telepresence projects, between libraries (Bibliolab) and between venues (Scènes ouvertes). Metalab’s partners include artists, university researchers, schools, designers and private companies.

Metalab Software

  • Immersive Space Editing and Prototyping: In situ editing (EiS)

  • Multi-projector video mapping for all types of surfaces: Splash

  • Calibration of projectors for immersive space: Calimiro

  • Sound spatialization for all types of speaker systems: SATIE

  • Sound spatialization by acoustic simulation: vaRays

  • Pose estimation of multiple people in an immersive space: Spook

  • Telepresence and multichannel transmission in low latency: Switcher

  • Sharing of all types of data streams between applications: Shmdata

Deep learning for Pose Estimation and Action Detection (closed)

Internship title

Deep Learning for Pose Estimation and Action Detection

Objective

Live audience interaction within art installations has been a long time interest for artists and creators. Using cameras and machine learning methods, the Metalab is developing a tool to add an interactive component to artistic experiences proposed by creators. This interactive component is based on detection of the audience's position and key points of the body, allowing real-time interaction with visual and sound elements.

The objective of this internship is to enhance the Metalab's interactive proposition by adding action detection that will allow user actions to control, influence, and transform visual and sound elements; for example, stimulating particle movement or modifying the shape or position of an object through physical gestures.

Tasks

With the support of the Metalab team:

  • Exploration of pose estimation and action detection techniques for our use case

  • Integration of tools based on artificial intelligence and machine learning algorithms

  • Work with video streams from RGB cameras (webcams, traditional cameras, as well as industrial cameras)

  • Participate in the lifecycle of the lab: scrums, code reviews, etc.

  • Documentation of the work and ensuring its reproducibility

Work environment

  • Python

  • Tensorflow, OpenCV

  • JIRA / Confluence

  • GitLab

  • Linux, Free and Open Source Software (FOSS)

Development of a graphic interface for a video mapping software (closed)

Internship title

Development of a graphic interface for a video mapping software

Objective

The graphical interface is often the first layer between the user and software and determines whether the software is easily usable or not. This internship focuses on this essential component with the overall goal to revisit the current interface of Splash, our video mapping software, and propose a new graphical interface targeting both beginner and intermediate users.

Splash is used in a high performance context for the [SAT] productions in the Satosphere. In this context, it handles both the mapping of the space and the rendering to the 8 projectors in the dome.

The objective of this internship is to develop a graphical interface that adresses the common use cases of Splash such as live performances with an artist, dome mapping while touring, immersive conference presentations, festivals.

Tasks

With the support of the Metalab team:

  • Analyze the use cases

  • Suggest one or more interface models

  • Develop and evaluate prototypes

  • Participate in the lifecycle of the lab: scrums, code reviews, etc.

  • Documentation of the work and ensuring its reproducibility

Work environment

  • C++ / Python

  • JIRA / Confluence

  • GitLab

  • Linux, Free and Open Source Software (FOSS)

Strategies for diverse, open and inclusive open source software communities

Internship title

Strategies for an open, diverse and inclusive software community - the case of Metalab open source softwares

Goal

As part of the Metalab's ongoing research projects on immersion and teleprescence, our team creates, develops and maintains a suite of free and open source software tools. We promote the use of this toolbox for a variety of projects.

We wish to enable the creation and the development of an open, diverse, inclusive community surrounding this toolbox and its use, supporting community contributions to our software.

The goals of this internship are two-fold: first, to assess available strategies to help ensure an open and inclusive communtiy setting. The second step will be an analysis of their adequacy for the Metalab software projects.

These strategies will be part of a research report.

Tasks

With support from the Metalab team, exploration of the following questions:

  • How can we enable the emergence of an open, diverse and inclusive software community?

  • Which tangible actions could be taken by the Metalab to foster community participation?

With support from the Metalab team, conduct a research project including:

  • Exploration and documentation of community practices followed in arts communities

  • Exploration and documentation of community practices followed in open source software communities

  • Recommandations for community facilitation tools

  • Recommandations for the Metalab to adopt such tools and practices

Work environment

Our environment promotes team work and the adoption of open source software, including:

  • Scrums and code review

  • GitLab, Python, Markdown, Bash

  • Linux, Free and Open Source Software (FOSS)

  • LaTeX