Past research

In my RL research, I was particularly interested in formulating “correct” optimization objectives. A task can be easy to describe in words, yet surprisingly difficult to translate into a reward function. Getting as close as possible to a goal, for example, is not quite the same as accumulating rewards along the way.

In my work on maximum-reward RL, I studied an alternative: optimize the best reward obtained during an episode. The policy then needs to know the maximum reward seen so far. By adding this to the state, I derived a Bellman operator and policy gradient results that let us use modern RL algorithms. I evaluated the approach on maze navigation and robotic manipulation tasks, and presented the work at ICML 2024.

Constraints were another motivation. Keeping the next action safe is not enough if it leaves the agent unable to avoid a collision later. This led me to ask whether we could constrain the highest cost along a trajectory, rather than its accumulated cost.

My PhD was about formulating the right problem: translating a real-world task into an optimization problem that is both tractable and aligned with what we actually want to achieve. Under uncertainty, the formulation depends on what we know about the system and what data we have. This was the common thread connecting my work on mathematical optimization, decision-focused learning, and reinforcement learning.

In electric-vehicle charging, we knew the grid model but only had partial information about where vehicles were connected. Explicitly considering every possible assignment would make the problem too large. I developed surrogate grid formulations that let us make charging decisions without modeling each possible location, balancing computational efficiency with solution quality.

Decision-focused learning takes the next step: some parameters are unknown, but we have data to predict them. The question is how to formulate the training problem. A navigation app, for example, should provide a good route; accurate predictions of travel times are only a means to that end. I was interested in training models for the quality of the decisions they produce, while using our knowledge of the optimization problem’s structure.

Choosing the right objective is only part of the problem: we also need to be able to learn from it. In convex optimization, changes in a prediction can leave the optimal decision unchanged, giving a zero gradient. I studied the geometry behind this and developed reformulations using quadratic approximation and local constraint smoothing to provide useful gradients for learning.

No active constraints

Free to move

One active constraint

Can slide along the edge

Two active constraints

Can remain pinned at the corner

At Nokia Bell Labs, I also applied decision-focused learning to time-series prediction for home energy management. Across these projects, the aim was to find formulations that preserve what matters about the task while making it possible to solve and learn from it.

Before my PhD, I worked at the Max Delbrück Center in Berlin, applying deep learning to study naked mole-rat vocalizations.

A large part of my work was making these recordings usable for analysis: detecting calls, extracting acoustic features, and classifying vocalizations at scale. The automated pipeline reduced manual annotation effort by approximately 80%.

My machine-learning analyses contributed to evidence that naked mole-rat colonies have distinct vocal dialects, leading to our publication in Science.