Deep learning for Pose Estimation and Action Detection

Internship title

Deep Learning for Pose Estimation and Action Detection

Objective

Live audience interaction within art installations has been a long time interest for artists and creators. Using cameras and machine learning methods, the Metalab is developing a tool to add an interactive component to artistic experiences proposed by creators. This interactive component is based on detection of the audience's position and key points of the body, allowing real-time interaction with visual and sound elements.

The objective of this internship is to enhance the Metalab's interactive proposition by adding action detection that will allow user actions to control, influence, and transform visual and sound elements; for example, stimulating particle movement or modifying the shape or position of an object through physical gestures.

Tasks

With the support of the Metalab team:

  • Exploration of pose estimation and action detection techniques for our use case

  • Integration of tools based on artificial intelligence and machine learning algorithms

  • Work with video streams from RGB cameras (webcams, traditional cameras, as well as industrial cameras)

  • Participate in the lifecycle of the lab: scrums, code reviews, etc.

  • Documentation of the work and ensuring its reproducibility

Work environment

  • Python

  • Tensorflow, OpenCV

  • JIRA / Confluence

  • GitLab

  • Linux, Free and Open Source Software (FOSS)

Contact

Please send your CV to metalab-dir@sat.qc.ca

Development of a graphic interface for a video mapping software

Internship title

Development of a graphic interface for a video mapping software

Objective

The graphical interface is often the first layer between the user and software and determines whether the software is easily usable or not. This internship focuses on this essential component with the overall goal to revisit the current interface of Splash, our video mapping software, and propose a new graphical interface targeting both beginner and intermediate users.

Splash is used in a high performance context for the [SAT] productions in the Satosphere. In this context, it handles both the mapping of the space and the rendering to the 8 projectors in the dome.

The objective of this internship is to develop a graphical interface that adresses the common use cases of Splash such as live performances with an artist, dome mapping while touring, immersive conference presentations, festivals.

Tasks

With the support of the Metalab team:

  • Analyze the use cases

  • Suggest one or more interface models

  • Develop and evaluate prototypes

  • Participate in the lifecycle of the lab: scrums, code reviews, etc.

  • Documentation of the work and ensuring its reproducibility

Work environment

  • C++ / Python

  • JIRA / Confluence

  • GitLab

  • Linux, Free and Open Source Software (FOSS)

Contact

Please send your CV to metalab-dir@sat.qc.ca

Strategies for diverse, open and inclusive open source software communities

Internship title

Strategies for an open, diverse and inclusive software community - the case of Metalab open source softwares

Goal

As part of the Metalab's ongoing research projects on immersion and teleprescence, our team creates, develops and maintains a suite of free and open source software tools. We promote the use of this toolbox for a variety of projects.

We wish to enable the creation and the development of an open, diverse, inclusive community surrounding this toolbox and its use, supporting community contributions to our software.

The goals of this internship are two-fold: first, to assess available strategies to help ensure an open and inclusive communtiy setting. The second step will be an analysis of their adequacy for the Metalab software projects.

These strategies will be part of a research report.

Tasks

With support from the Metalab team, exploration of the following questions:

  • How can we enable the emergence of an open, diverse and inclusive software community?

  • Which tangible actions could be taken by the Metalab to foster community participation?

With support from the Metalab team, conduct a research project including:

  • Exploration and documentation of community practices followed in arts communities

  • Exploration and documentation of community practices followed in open source software communities

  • Recommandations for community facilitation tools

  • Recommandations for the Metalab to adopt such tools and practices

Work environment

Our environment promotes team work and the adoption of open source software, including:

  • Scrums and code review

  • GitLab, Python, Markdown, Bash

  • Linux, Free and Open Source Software (FOSS)

  • LaTeX

Contact

Please send your CV to metalab-dir@sat.qc.ca

Animation / Facial and body rigging for an immersive experience

Internship title

Animation / Facial and body rigging for an immersive experience

Objective

As part of its research efforts, the SAT's Metalab develops interactive tools for the detection of audience member's positions and poses, using videostreams from live events and installations.

In collaboration with the Metalab team, you will produce a demonstration of real-time capture of participants' facial expressions and body posture and its representation as an avatar in a virtual environment.

A [previous internship](https://sat.qc.ca/fr/nouvelles/workflow-de-capture-et-reproduction-de-visage-dans-blender) on this subject has laid the groundwork for this project.

Tasks

With the support of the Metalab team:

  • Experiment with avatars and their skeletons

  • Model and edit 3D avatars

  • Animate / rig 3D models of face and body

  • Participate in the lifecycle of the lab: scrums, code reviews, etc.

  • Documentation of the work and ensuring its reproducibility

Work environment

  • 3D modeling, editing, and processing

  • Python

  • Blender

  • JIRA / Confluence

  • GitLab

  • Linux, Free and Open Source Software (FOSS)

Contact

Please send your CV to metalab-dir@sat.qc.ca

SATIE additional spatializers

Internship title

Additional spatializer types for SATIE

Objective

SATIE is our in-house audio spatializer that aims to be flexible, robust and supports many different arrangements of speaker arrays. This internship aims at bringing more spatialization techniques to SATIE.

Tasks

  • Participate in the ideation
  • Review the literature that documents and discusses spatialization techniques
  • Develop and test spatializer plugins for SATIE (using the SuperCollider framework)
  • Participate in the development of a demo (video or live)
  • Participate in the life of the laboratory: scrums, code review, etc.
  • Document the work and ensure its reproducibility.

Context and software

  • Linux OS
  • Digital signal processing
  • SuperCollider, bash
  • 3D audio and ambisonics
  • VR/AR
  • Optional: Python

Contact

Please send your CV to metalab-dir@sat.qc.ca

IR, Calibration and Visualisation

Internship title

Impulse Response, Calibration and Visualisation of Audio

Objective

This interships aims to continue the development of recipes, approaches and tools around Impulse Response recording and visualisation. The purpose of this work is to identify some acoustic properties of a physical space to help with calibration of audio equipment.

Tasks

  • Participate in the ideation
  • Explore existing in-house and other tools for similar tasks
  • Experiment and develop or improve existing in-house tools or pipelines involving other tools
  • Participate in the development of a demo (video or live)
  • Participate in the life of the laboratory: scrums, code review, etc.
  • Document the work and ensure its reproducibility.

Context and software

  • Linux OS
  • Digital signal processing
  • Python (numpy), bash
  • 3D audio and ambisonics
  • VR/AR
  • Optional: SuperCollider

Contact

Please send your CV to metalab-dir@sat.qc.ca

Add directionality for sources to our audio ray tracing tool (vaRays)

Internship title

Add directionality for sources to our audio ray tracing tool (vaRays)

Objective

Our audio simulation engine, vaRays, uses ray tracing to simulate acoustics in virtual environments. The objective of this internship will be to propose and implement a method that will allow different possible energy distributions for audio sources in vaRays.

Tasks

With the help of the Metalab team:

  • Get acquainted with the vaRays code

  • Experiment with and propose different methods for the implementation of directionality for audio sources

  • Participate in the creation of demonstrations of those implementations

  • Document the work and ensure its reproducibility

Work environment

  • Scrums, code review, etc.

  • C++

  • GitLab, Linux, Free and Open Source Software (FOSS)

  • JIRA / Confluence

Contact

Please send your CV to metalab-dir@sat.qc.ca

Re-enforcing continuous integration in gitlab

Internship title

Re-enforcing continuous integration of Free and Open Source Software (FOSS) developped at the Metalab

Objective

As we want to offer more support for our tools, we need to implement this support in our CI in Gitlab. Multiple directions are possible: * Refine software packaging for better distribution * Support building for ARM architecture in CI (currently only x86) * Augment test coverage (Python, C++) * Investigate testing ARM architecture in CI

Tasks

With the help of the Metalab team: * Identify areas where tests / packaging / cross compilation are missing * Implement new tests * Implement cross-compilation on CI * Implement / refine packaging

Work environment

  • GitLab, Linux, Free and Open Source Software (FOSS)

  • C++ / Python / YAML

  • JIRA / Confluence

  • Docker, QEMU

  • NVidia Jetson, Raspberry Pi

Contact

Please send your CV to metalab-dir@sat.qc.ca

Audio C++/Python Programming for Technological Arts

Position

The Society for Arts and Technology [SAT] needs you for an immediate start at Metalab, the SAT research laboratory. Your role will be to participate in the research process: technology watch, ideation, development, implementation, communication and participation in partnership projects. You will participate in the evolution, code review and use of Metalab software. An experience of at least 5 years is desired.

Technical environment

  • Ubuntu, Linux, FOSS

  • Numpy, libeigen, liblo

  • Gcc, python3

  • Gitlab, JIRA, Confluence, Slack, Mozilla Hubs

Assets

  • SuperCollider

  • Spatial audio, ambisonics

  • Acoustics

  • Blender

  • GDB, cmake, clang

Work environment

Since 2002, the Metalab is the Society for Arts and Technology [SAT]’s research and development lab. The Metalab has a dual mission: to stimulate the emergence of new immersive experiences, and to make their design and authoring accessible to artists and creators through an ecosystem of free software.

The Metalabs research fields – telepresence, immersion, video mapping, and spatialized sound – are explored through research and production projects. The Metalab team is comprised of 8 to 15 people with various technical expertise including digital audio, computer graphics, networks, software development and multimedia integration. The Metalab welcomes interns in a variety of subjects, even outside the strict scope of the laboratory expertises.

The Metalab's works aims to cover the whole pipeline of immersive and distributed content production. The Metalab’s software addresses, among others, immersive in situ editing, video mapping, audio spatialization, group interaction and low latency transmission. They are designed as flexible tools, and supports Metalab’s research and partnerships. This software suite is used by the SAT during creative residencies and is at the heart of telepresence projects, between libraries (Bibliolab) and between venues (Scènes ouvertes). Metalab’s partners include artists, university researchers, schools, designers and private companies.

Metalab Software

  • Immersive Space Editing and Prototyping: In situ editing (EiS)

  • Multi-projector video mapping for all types of surfaces: Splash

  • Calibration of projectors for immersive space: Calimiro

  • Sound spatialization for all types of speaker systems: SATIE

  • Sound spatialization by acoustic simulation: vaRays

  • Group interaction of people in an immersive space: Spook

  • Telepresence and multichannel transmission in low latency: Switcher

  • Sharing of all types of data streams between applications: Shmdata

Contacts

Send a CV to metalab-dir@sat.qc.ca

Linux C++/Python Programming for Technological Arts

Position

The Society for Arts and Technology [SAT] needs you for an immediate start at Metalab, the SAT research laboratory. Your role will be to participate in the research process: technology watch, ideation, development, implementation, communication and participation in partnership projects. You will participate in the evolution, code review and use of Metalab software. An experience of at least 5 years is desired.

Technical environment

  • Ubuntu, Linux, FOSS

  • Gcc, python3, cmake

  • GDB, clang

  • Raspberry Pi, NVidia jetson

  • Gitlab CI, Docker, AWS, Azure

  • Gitlab, JIRA, Confluence, Slack, Mozilla Hubs

Assets

  • Real-time audio-visual

  • TCP/IP

  • Cross compilation

  • Kubernetes

Work environment

Since 2002, the Metalab is the Society for Arts and Technology [SAT]’s research and development lab. The Metalab has a dual mission: to stimulate the emergence of new immersive experiences, and to make their design and authoring accessible to artists and creators through an ecosystem of free software.

The Metalabs research fields – telepresence, immersion, video mapping, and spatialized sound – are explored through research and production projects. The Metalab team is comprised of 8 to 15 people with various technical expertise including digital audio, computer graphics, networks, software development and multimedia integration. The Metalab welcomes interns in a variety of subjects, even outside the strict scope of the laboratory expertises.

The Metalab's works aims to cover the whole pipeline of immersive and distributed content production. The Metalab’s software addresses, among others, immersive in situ editing, video mapping, audio spatialization, group interaction and low latency transmission. They are designed as flexible tools, and supports Metalab’s research and partnerships. This software suite is used by the SAT during creative residencies and is at the heart of telepresence projects, between libraries (Bibliolab) and between venues (Scènes ouvertes). Metalab’s partners include artists, university researchers, schools, designers and private companies.

Metalab Software

  • Immersive Space Editing and Prototyping: In situ editing (EiS)

  • Multi-projector video mapping for all types of surfaces: Splash

  • Calibration of projectors for immersive space: Calimiro

  • Sound spatialization for all types of speaker systems: SATIE

  • Sound spatialization by acoustic simulation: vaRays

  • Group interaction of people in an immersive space: Spook

  • Telepresence and multichannel transmission in low latency: Switcher

  • Sharing of all types of data streams between applications: Shmdata

Contacts

Send a CV to metalab-dir@sat.qc.ca