This job might no longer be available.

Research Internship (F/M/NB) - Learning Human-like Controllable Bots from Traces by Offline Reinforcement Learning

Job Description

Programming good bots in video games is still an open challenge. The usual way is to manually define a behavior tree that encodes the way the bot behaves together with the conditions that control when the agent has to switch from one behavior to another. This methodology has several drawbacks: it is very time consuming and limited to behaviors that we can code, resulting in unrealistic behaviors that decrease the quality of the games. Let us consider for instance that one wants to develop a bot in a first-person shooter that behaves ‘like a human’, this is very difficult to achieve with behavior trees since we don’t have a clear understanding of how humans are playing, and we are thus not able to mimic this behavior.

As a solution, reinforcement learning seems to be the natural candidate to automatically discover efficient bots. Algorithms in this domain have recently achieved strong performance in games (starcraft, chess, go, …) and are natural candidates for bots. But they also suffer of many drawbacks. The main one is that they usually rely on a reward function that one wants to optimize. In the previous example of ‘human-like bot’, there is not clear reward function defining the expected behavior. Indeed, each player is certainly optimizing their own function which is completely unknown.

We propose to address this problem differently, by exploiting game traces captured when real players are playing the game. This internship aims at defining new methods able to discover realistic bots from data instead of being guided by some reward functions or prior information about the nature of the game.

As a first step, we expect to develop models that will learn not only one behavior, but multiple and diverse behaviors that capture the continuum of human play styles. It will be related to imitation learning with the objective to learn simultaneously imitation policies guided by a latent information that capture the style of the player, the policy and the latent space being built together. If we have already made first steps in this direction based on the use of decision transformers, we plan to push our effort one step further by applying this family of models to realistic data extracted from video games.

As a second step, we expect to augment our methods with the ability to build a meaningful controllable space. Indeed, if learning a latent space of styles allows one to define different styles through different vectors in this space, it cannot be used to understand what are the style dimensions that we control. It thus makes these approaches difficult to use by designers that wants to define interesting behaviors from a gameplay perspective. To overcome this limit, we propose to enforce the model to map the latent space dimensions with meaningful information. As an example, one meaning full dimension is the play level of the bot which can be extracted directly from the dataset.

The internship will be executed using the set of tools developed in La Forge which include realistic multiplayer 3D games made specifically for research, large datasets of traces and learning algorithms deployed on GPU clusters.

Qualifications

You are a last year student of an engineering school or a university research master;

You have solid knowledge in mathematics and computer science

You have skills in machine learning, deep learning, or reinforcement learning, and have mobilized them using suitable Python libraries

Your level of English allows you to work in an international team and to communicate easily with non-French speakers.

Skills and competencies show up in different forms and can be based on different experiences, that's why we strongly encourage you to apply even though you may not have all the requirements listed above.

Additional Information

Process:

· Phone Interview

· Interview(s) with our internal teams

· Final interview with the project manager

If your application is not retained, you will receive a negative answer.

At Ubisoft, you can come as you are. We embrace diversity in all its forms. We’re committed to fostering a work environment that is inclusive and respectful of all differences, we value diversity at our company and do not discriminate on the basis of race, ethnicity, religion, gender, sexual orientation, age or disability status. All personal informations will be treated as confidential according to the Employment Equity act

Company Description

About Ubisoft

Ubisoft’s 20,000 team members, working across more than 30 countries around the world, are bound by a common mission to enrich players’ lives with original and memorable gaming experiences. Their commitment and talent have brought to life many acclaimed franchises such as Assassin’s Creed, Far Cry, Watch Dogs, Just Dance, Rainbow Six, and many more to come. Ubisoft is an equal opportunity employer that believes diverse backgrounds and perspectives are key to creating worlds where both players and teams can thrive and express themselves. If you are excited about solving game-changing challenges, cutting edge technologies and pushing the boundaries of entertainment, we invite you to join our journey and help us create the unknown.

Ubisoft Bordeaux

Founded in September 2017, Ubisoft Bordeaux works with passion on the biggest AAAA’s game in order to offer the best gaming experiences to our players. Today, the studio has more than 400 talents, from 15 different nationalities, who work on licenses such as Assassin's Creed, Beyond Good & Evil 2, plus other unannounced free-to-play games. We are also working on exciting technologies with the Anvil team, Online services teams and with La Forge who seek to validate the value of technological innovations.