Fast Forwarding and Simpoint support in MARSS

One of the most requested feature in MARSS has been fast-forwarding N instructions before starting simulation. Recently we have developed some hooks into QEMU's translation logic to count number of instructions emulated. With this logic we are now able to support fast-forwarding and simpoint in MARSS. Both of these features are available in features branch right now.

Fast-Forwarding
Following new simconfig options are added for fast-forwarding support in MARSS.

• -fast-fwd-insns N: Fast-forward N number of total instructions. This includes user level and kernel level instruction across all emulated CPUs. After specified limit is reached it will switch to simulation mode.
• -fast-fwd-user-insns N: Fast-forward N number of user level instructions. This mode will emulate kernel level instructions but doesn't count them. After specified user level instructions are executed it will switch to simulation mode and it will simulate both user and kernel level instructions.
• -fast-fwd-checkpoint CHK_NAME: Create a checkpoint named 'CHK_NAME' after fast-forwarding specified amount of instructions. It will kill simulation instance after a checkpoint is created.

As mentioned above the default behavior is to switch to simulation mode after specified instructions are emulated. Because of non-deterministic behavior of kernel level execution in emulation mode we also support creating a checkpoint after fast-forwarding so users can use fast-forwarding only once and then rely on checkpoint to start simulation at specific RIP everytime.

For multicore emulation/simulation, fast-forwarding specified amount of instruction is little complicated because many times only few of CPUs are executing any code and others are in idle mode. At first MARSS equally divides total number of instructions to emulate and allocate them to all CPUs in machine. When part of those CPUs are in idle mode then MARSS will re-allocate portion of remaining instruction count from idle CPUs to non-idle CPUs to reach to specified instructions limit.

Simpoint
Simpoints has been one of the most used and reliable method in computer-architecture research to simulate part of applications that is representative of full application run. We decided to implement support for Simpoints to evaluate performance of different applications with real-hardware, but more on that later. MARSS uses simpoint file to create checkpoints after specified instructions are emulated. To create checkpoints based on simpoints and how to use weights file and mstats.py to calculate weighted IPC please refer to wiki page on Simpoints.

Counting Emulated Instructions
QEMU's emulation engine - TCG is designed to be fast which first convert instruction into list of micro-instructions and then convert these micro-instructions into binary buffer that can be executed and re-executed without any change. This makes little bit difficult to directly count the number of instructions emulated.

Recently we developed support for Simpoints in which we need to count number of instructions emulated and create a checkpoint after specific number of instructions. During this development we figured out a way to add hooks into each translated binary block to count number of instructions emulated at run-time. We added a counter to each CPU context which is set to number of instructions a CPU context is allowed to execute. In each translated block we added a hook that decrements CPU context's instruction counter by number of instructions in the block. When the counter reaches to 0 we switch to simulation mode or create a checkpoint based on user's configuration options.

Synchronizing multiple simulation instances

Couple of months ago we introduced a new feature 'syncing multiple simulations instances' (available in 'features' branch right now). This feature allows users to run multiple instances of Marss that synchronize at given number of clock ticks. This blog post talks about what was the reason to implement such a feature.

Super-Duper fast Server Issue

We were running multiple instances of Marss on same machine, one running server and other running clients. Marss simulation speed varies a lot based on type of load so we noticed that when we run server and client together they drift apart in terms of simulated clocks as shown in the graph below.

As shown in the graph after 500 seconds of simulation the 'server' has executed around 80mil cycles where as 'client' has executed only 40mil cycles, a gap of 40+ million of cycles. This gaps keeps increasing as we keep running the simulations. Due to this high difference between two simulations it seems that 'server' is running with more than twice the speed as 'client' and all the request timing measured by 'client' are unrealistic as some of them posted request response within half nano-second. In real life the difference between two machines do exists but not this much - high end servers mostly run around 3 to 3.2 GHz and normal clients (laptops/desktop machines) run around 2 to 2.8 GHz speed. So due to this behavior our study of server and client benchmarks was flawed and we needed to fix the relative speed of each simulation instance.

Synchronizing Simulations

Once we realized the issue with this setup we looked into multiple options to keep the multiple instances in sync so the clock don't drift apart too much as we run simulations for long time. First thing I tried was to limit the maximum number of cycles to execute in given time frame. The issue with this technique was when each instance is capable of running faster even then we were limiting the speed and our total simulation time was increased by more than 2x.

So I decided to give SysV Semaphores a try as described in previous blog post to sync multiple processes using semaphores as barrier. The implementation was very simple, each instance is allowed to execute fix number of cycles between each barrier. So we ran some simulations with different interval size and found out that 200K cycles barrier was good enough to reduce the clock drift between each instance while minimizing the effect on simulation speed.

Here is the resulting graph after running simulations with sync feature. As shown in the image now both the 'server' and 'clients' execute with same frequency so request time measured by 'clients' are now realistic.

To use this feature provide '-sync N' simconfig option to each simulation instance that you want to run in synchronization. Here the N is number of cycles to execute between each sync.