Jason's Website

Visualizing Spectre with gem5

2018-06-01T00:00:00-05:00

Spectre and Meltdown took much of our community by surprise. I personally found these attacks fascinating because they didn't rely on a bug in any particular hardware implementation, but leveraged undefined behavior. Specifically, Spectre and Meltdown can exfiltrate potentially secret memory data by detecting the effects of speculative instructions that are later squashed.

Very cool!

Out of order processors are very complex. It would make it easier to understand exactly what causes speculation attacks like Spectre and Meltdown if we had a way to visualize the attacks. Luckily, gem5 already has a way to view the details of it's out of order CPU's pipeline.

o3 pipeline view example

The image above was created using the O3 pipeline viewer that is included with gem5. In this post, I'll explain how to use the O3 pipeline viewer and how to generate images like the above. There is also a new project which makes it easier to navigate large pipeline traces and it quite useful for comparing different pipeline designs: Konata created by Ryota Shioya. Ryota gave a presentation on Konata at a recent Learning gem5 tutorial. You can find the pdf of his presentation here. Konata is a cool tool that's written in javascript and Ryota describes it as "Google maps for an out of order pipeline".

Running Spectre

The first step to visualizing what is going on in the pipeline during a Spectre attack is getting proof of concept exploit code. I used the code that was posted to a github gist by Erik August soon after the attack was announced. You can get that code here: https://gist.github.com/ErikAugust/724d4a969fb2c6ae1bbd7b2a9e3d4bb6.

First, you need to compile the proof of concept code on your native machine (note: I'll be using x86 for all of my examples).

gcc spectre.c -o spectre -static

I used gcc 7.2 (the default on Ubuntu 17.10) for my tests, and you may want to do the same. Below I discuss the effects different compilers have on the Specre attack. For instance, if you use clang instead you may not be able to reproduce the Spectre attack in gem5.

docker run --rm -v $PWD:$PWD -w $PWD powerjg/gem5-build-gcc48 gcc spectre.c -o spectre -static -std=c99

My native machine is still vulnerable to Spectre so when I run the binary generated above, I get the following output.

Reading 40 bytes:
Reading at malicious_x = 0xffffffffffdd76c8... Success: 0x54=’T’ score=2
Reading at malicious_x = 0xffffffffffdd76c9... Success: 0x68=’h’ score=2
Reading at malicious_x = 0xffffffffffdd76ca... Success: 0x65=’e’ score=2
Reading at malicious_x = 0xffffffffffdd76cb... Success: 0x20=’ ’ score=2
Reading at malicious_x = 0xffffffffffdd76cc... Success: 0x4D=’M’ score=2
Reading at malicious_x = 0xffffffffffdd76cd... Success: 0x61=’a’ score=2
Reading at malicious_x = 0xffffffffffdd76ce... Success: 0x67=’g’ score=2
Reading at malicious_x = 0xffffffffffdd76cf... Success: 0x69=’i’ score=2
Reading at malicious_x = 0xffffffffffdd76d0... Success: 0x63=’c’ score=2
Reading at malicious_x = 0xffffffffffdd76d1... Success: 0x20=’ ’ score=2
Reading at malicious_x = 0xffffffffffdd76d2... Success: 0x57=’W’ score=2
Reading at malicious_x = 0xffffffffffdd76d3... Success: 0x6F=’o’ score=2
Reading at malicious_x = 0xffffffffffdd76d4... Success: 0x72=’r’ score=2
Reading at malicious_x = 0xffffffffffdd76d5... Success: 0x64=’d’ score=2
Reading at malicious_x = 0xffffffffffdd76d6... Success: 0x73=’s’ score=2
Reading at malicious_x = 0xffffffffffdd76d7... Success: 0x20=’ ’ score=2
Reading at malicious_x = 0xffffffffffdd76d8... Success: 0x61=’a’ score=2
Reading at malicious_x = 0xffffffffffdd76d9... Success: 0x72=’r’ score=2
Reading at malicious_x = 0xffffffffffdd76da... Success: 0x65=’e’ score=2
Reading at malicious_x = 0xffffffffffdd76db... Success: 0x20=’ ’ score=2
Reading at malicious_x = 0xffffffffffdd76dc... Success: 0x53=’S’ score=2
Reading at malicious_x = 0xffffffffffdd76dd... Success: 0x71=’q’ score=2
Reading at malicious_x = 0xffffffffffdd76de... Success: 0x75=’u’ score=2
Reading at malicious_x = 0xffffffffffdd76df... Success: 0x65=’e’ score=2
Reading at malicious_x = 0xffffffffffdd76e0... Success: 0x61=’a’ score=2
Reading at malicious_x = 0xffffffffffdd76e1... Success: 0x6D=’m’ score=2
Reading at malicious_x = 0xffffffffffdd76e2... Success: 0x69=’i’ score=2
Reading at malicious_x = 0xffffffffffdd76e3... Success: 0x73=’s’ score=2
Reading at malicious_x = 0xffffffffffdd76e4... Success: 0x68=’h’ score=2
Reading at malicious_x = 0xffffffffffdd76e5... Success: 0x20=’ ’ score=2
Reading at malicious_x = 0xffffffffffdd76e6... Success: 0x50=’P’ score=9 (second best: 0x06 score=2)
Reading at malicious_x = 0xffffffffffdd76e7... Success: 0x73=’s’ score=2
Reading at malicious_x = 0xffffffffffdd76e8... Success: 0x73=’s’ score=2
Reading at malicious_x = 0xffffffffffdd76e9... Success: 0x69=’i’ score=2
Reading at malicious_x = 0xffffffffffdd76ea... Success: 0x66=’f’ score=2
Reading at malicious_x = 0xffffffffffdd76eb... Success: 0x72=’r’ score=2
Reading at malicious_x = 0xffffffffffdd76ec... Success: 0x61=’a’ score=2
Reading at malicious_x = 0xffffffffffdd76ed... Success: 0x67=’g’ score=2
Reading at malicious_x = 0xffffffffffdd76ee... Success: 0x65=’e’ score=2
Reading at malicious_x = 0xffffffffffdd76ef... Success: 0x2E=’.’ score=2

Running Spectre in gem5

To find out if gem5's out of order CPU implementation is vulnerable to Spectre, we need to run the code in gem5. The simplest and fastest way to do this is by running in gem5's syscall-emulation (SE) mode. In SE mode we won't be modeling an OS or any user-mode to kernel-mode interaction, but this okay for Spectre since this proof of concept code is all in user-mode. If we were investigating Metldown, we would have to use full-system (FS) mode since Meltdown specifically allows user-mode processes to read data that should only be accessible in kernel mode.

So, when running something in gem5, the first step is to create a Python runscript since this is the "interface" to gem5. For this example, what we need is a system with one CPU, an L1 cache, and memory. For simplicity, I'm going to modify one of the existing script, specifically the two_level.py script from the Learning gem5 book.

In the file gem5/configs/learning_gem5/part1/two_level.py, I simply changed the CPU from TimingSimpleCPU() to DerivO3CPU(branchPred=LTAGE()). I also set the O3CPU to use the LTAGE branch predictor instead of the default tournament branch predictor. It's important to use the LTAGE branch predictor as better branch predictors actually make Spectre easier to exploit as discussed further below.

Now, we simply need to build gem5 and run it.

scons -j8 build/X86/gem5.opt

build/X86/gem5.opt configs/learning_gem5/part1/two_level.py spectre

And, the output that I get is the following, just like above when I ran the spectre natively. Note: If you use GCC 4.8 it executes much faster than 7.2 (the default on Ubuntu 17.10).

gem5 Simulator System.  http://gem5.org
gem5 is copyrighted software; use the --copyright option for details.

gem5 compiled May 10 2018 09:40:08
gem5 started May 24 2018 11:21:16
gem5 executing on palisade, pid 27173
command line: build/X86/gem5.opt configs/learning_gem5/part1/two_level.py spectre

Global frequency set at 1000000000000 ticks per second
warn: DRAM device capacity (8192 Mbytes) does not match the address range assigned (512 Mbytes)
0: system.remote_gdb: listening for remote gdb on port 7000
Beginning simulation!
info: Entering event queue @ 0.  Starting simulation...
warn: readlink() called on '/proc/self/exe' may yield unexpected results in various settings.
      Returning '/home/jlp/Code/gem5/spectre-vis/spectre'
info: Increasing stack size by one page.
warn: ignoring syscall access(...)
Reading 40 bytes:                 tput cols
Reading at malicious_x = 0xffffffffffdd76c8... Success: 0x54=’T’ score=2
Reading at malicious_x = 0xffffffffffdd76c9... Success: 0x68=’h’ score=2
Reading at malicious_x = 0xffffffffffdd76ca... Success: 0x65=’e’ score=2
Reading at malicious_x = 0xffffffffffdd76cb... Success: 0x20=’ ’ score=2
Reading at malicious_x = 0xffffffffffdd76cc... Success: 0x4D=’M’ score=2
Reading at malicious_x = 0xffffffffffdd76cd... Success: 0x61=’a’ score=2
Reading at malicious_x = 0xffffffffffdd76ce... Success: 0x67=’g’ score=2
Reading at malicious_x = 0xffffffffffdd76cf... Success: 0x69=’i’ score=2
Reading at malicious_x = 0xffffffffffdd76d0... Success: 0x63=’c’ score=2
Reading at malicious_x = 0xffffffffffdd76d1... Success: 0x20=’ ’ score=2
Reading at malicious_x = 0xffffffffffdd76d2... Success: 0x57=’W’ score=2
Reading at malicious_x = 0xffffffffffdd76d3... Success: 0x6F=’o’ score=2
Reading at malicious_x = 0xffffffffffdd76d4... Success: 0x72=’r’ score=2
Reading at malicious_x = 0xffffffffffdd76d5... Success: 0x64=’d’ score=2
Reading at malicious_x = 0xffffffffffdd76d6... Success: 0x73=’s’ score=2
Reading at malicious_x = 0xffffffffffdd76d7... Success: 0x20=’ ’ score=2
Reading at malicious_x = 0xffffffffffdd76d8... Success: 0x61=’a’ score=2
Reading at malicious_x = 0xffffffffffdd76d9... Success: 0x72=’r’ score=2
Reading at malicious_x = 0xffffffffffdd76da... Success: 0x65=’e’ score=2
Reading at malicious_x = 0xffffffffffdd76db... Success: 0x20=’ ’ score=2
Reading at malicious_x = 0xffffffffffdd76dc... Success: 0x53=’S’ score=2
Reading at malicious_x = 0xffffffffffdd76dd... Success: 0x71=’q’ score=2
Reading at malicious_x = 0xffffffffffdd76de... Success: 0x75=’u’ score=2
Reading at malicious_x = 0xffffffffffdd76df... Success: 0x65=’e’ score=2
Reading at malicious_x = 0xffffffffffdd76e0... Success: 0x61=’a’ score=2
Reading at malicious_x = 0xffffffffffdd76e1... Success: 0x6D=’m’ score=2
Reading at malicious_x = 0xffffffffffdd76e2... Success: 0x69=’i’ score=2
Reading at malicious_x = 0xffffffffffdd76e3... Success: 0x73=’s’ score=2
Reading at malicious_x = 0xffffffffffdd76e4... Success: 0x68=’h’ score=2
Reading at malicious_x = 0xffffffffffdd76e5... Success: 0x20=’ ’ score=2
Reading at malicious_x = 0xffffffffffdd76e6... Success: 0x4F=’O’ score=2
Reading at malicious_x = 0xffffffffffdd76e7... Success: 0x73=’s’ score=2
Reading at malicious_x = 0xffffffffffdd76e8... Success: 0x73=’s’ score=2
Reading at malicious_x = 0xffffffffffdd76e9... Success: 0x69=’i’ score=2
Reading at malicious_x = 0xffffffffffdd76ea... Success: 0x66=’f’ score=2
Reading at malicious_x = 0xffffffffffdd76eb... Success: 0x72=’r’ score=2
Reading at malicious_x = 0xffffffffffdd76ec... Success: 0x61=’a’ score=2
Reading at malicious_x = 0xffffffffffdd76ed... Success: 0x67=’g’ score=2
Reading at malicious_x = 0xffffffffffdd76ee... Success: 0x65=’e’ score=2
Reading at malicious_x = 0xffffffffffdd76ef... Success: 0x2E=’.’ score=2
Exiting @ tick 113568969000 because exiting with last active thread context

Visualizing the out of order pipeline

To generate pipeline visualizations, we first need to generate a trace file of all of the instructions executed by the out of order CPU. To create this trace, we can use the O3PipeView debug flag.

Now, the trace for the O3 CPU can be very large up to many GBs. When creating this trace, you need to be careful to create the smallest trace possible. Also, it's important to dump the trace to a file and not to stdout, which is the default when using debug flags. You can redirect the trace to a file by using the --debug-file option to gem5.

To create the trace file, I used the following methodology:

Start running spectre in gem5, then hit ctrl-c after the first couple of letters. At this point, I wrote down the tick which gem5 exited (13062347000).
Run gem5 with the debug flag O3PipeView enabled.
Watch the output and kill gem5 with ctrl-c after two more letters appeared than in step 1.

To generate the trace, I ran the following command. Note: you may have a different value for when to start the debugging trace. Also note: when producing the trace gem5 will run much slower.

build/X86/gem5.opt --debug-flags=O3PipeView --debug-file=pipeview.txt --debug-start=13062347000 configs/learning_gem5/part1/two_level.py spectre

My tracefile (pipeview.txt) was 600 MB for catching just two letters in the output.

Now, we can process this file to generate the visualization with a script: util/o3-pipeview.py. This script requires the path to the file that contains the output generated with the O3PipeView debug flag. Above, we put the output into the file pipeview.txt, and this file was created in the default output directory of gem5 (m5out/).

util/o3-pipeview.py --store_completions m5out/pipeview.txt --color -w 150

In the above command, I wanted to see when the stores completed (--store_completions) and specified to use color (--color) in the output and use a width of 150 characters (-w 150). Processing a large file like this one of 600 MB may take a few minutes. The output will be in a file called o3-pipeview.out in the current working directory.

You can view this file with less -r o3-pipeview.out. You may want to use the -S option with less if your terminal is less than 150 characters wide (or whatever width value you used). Below is a screenshot of the top of my trace.

Understanding the O3 pipeline viewer

o3 pipeline view example

The above image details how to interpret the output from the pipeline viewer. Each . or = represents one cycle of time, which moves from left to right. The "tick" column shows the tick of the leftmost . or =. = is used to mark the instructions that were later squashed. The address of the instruction (and the micro-op number) as well as the disassembly is also shown. The sequence number can be ignored as it is always monotonically increasing and is the total order of every dynamic instruction. Finally, each stage of the O3 pipeline is shown with a different letter and color.

Digging deeper into Spectre

First, let's examine the actual instructions that are executed during the Spectre attack. The vulnerability is in the victim_function in spectre.c.

void victim_function(size_t x) {
  if (x < array1_size) {
    temp &= array2[array1[x] * 512];
  }
}

When this is compiled and then dumped with objdump, we get the following instructions that will be executed. Your code my be slightly different, especially the exact addresses of each instruction, depending on the version of the compiler and other system-specific configurations.

# NOTE: the movzbl below is MOVZX_B_R_M in gem5.
# it is implemented with the following microcode.
#    ld t1, seg, sib, disp, dataSize=1
#    zexti reg, t1, 7
#
000000000040105e <victim_function>:
  40105e:   55                      push   %rbp
  40105f:   48 89 e5                mov    %rsp,%rbp
  401062:   48 89 7d f8             mov    %rdi,-0x8(%rbp)
  401066:   8b 05 14 f0 2b 00       mov    0x2bf014(%rip),%eax # 6c0080 <array1_size> load array1_size (first time is always a miss)
  40106c:   89 c0                   mov    %eax,%eax
  40106e:   48 3b 45 f8             cmp    -0x8(%rbp),%rax  # if (x < array1_size) rax is array1_size, -8(%rbp) is x
  401072:   76 2b                   jbe    40109f <victim_function+0x41> # if (x < array1_size)
  401074:   48 8b 45 f8             mov    -0x8(%rbp),%rax # load x from the stack into rax
  401078:   48 05 a0 00 6c 00       add    $0x6c00a0,%rax  # calculate array1 offset (x+array1)
  40107e:   0f b6 00                movzbl (%rax),%eax # load array1[x]
  401081:   0f b6 c0                movzbl %al,%eax    # zero extend to 32 bits
  401084:   c1 e0 09                shl    $0x9,%eax   # multiply by 512
  401087:   48 98                   cltq               # sign-extend eax
  401089:   0f b6 90 80 1d 6c 00    movzbl 0x6c1d80(%rax),%edx  # load array2[array1[x]*512] **** This is the magic!
  401090:   0f b6 05 e9 0c 2e 00    movzbl 0x2e0ce9(%rip),%eax        # 6e1d80 <temp> Load temp.
  401097:   21 d0                   and    %edx,%eax
  401099:   88 05 e1 0c 2e 00       mov    %al,0x2e0ce1(%rip)        # 6e1d80 <temp>
  40109f:   5d                      pop    %rbp
  4010a0:   c3                      retq

Now, we can search for the instruction that we care about in the trace. In this case, we want to find a time where the movzbl at address 0x401089 is executed speculatively. When searching through the pipeline viewer (use \ in less), we're looking for a time where the load completes for the instruction at 0x401089 and it is later squashed (surrounded by =). An example is shown below.

annotated O3 pipeline view of spectre

The image above is from my presentation at Hardware and Architectural Support for Security and Privacy (HASP) 2018.

What we see in this image is that the instruction at 0x401066 causes a cache miss (there is a long time between when the load is issued and the data is returned from memory). Since the load of array1_size was a cache miss, the jump at 0x401072 is speculated to be not taken (incorrectly). This causes the following instructions to be executed speculatively, and, eventually, squashed.

The key thing in this trace that is the Spectre vulnerability is that the load for the instruction at 0x40107e, which loads secret data happens during the mis-speculated instructions. Then, this data is loaded into the registers and operated on (instruction 0x401084). Finally, the load at address 0x401089 is executed and loads the value from memory that is dependent on the secret data loaded previously. Thus, we can later probe the cache to retrieve the secret data.

Effects of compilers

As previously mentioned, the specific compiler version and compiler options have a significant effect on the attack. Below are two traces, one from GCC 7.2 and one from clang 4.0.

GCC 7.2

void victim_function(size_t x) {
  400b2d:       55                      push   %rbp
  400b2e:       48 89 e5                mov    %rsp,%rbp
  400b31:       48 89 7d f8             mov    %rdi,-0x8(%rbp)
  if (x < array1_size) {
  400b35:       8b 05 c5 c5 2c 00       mov    0x2cc5c5(%rip),%eax        # 6cd100 <array1_size>
  400b3b:       89 c0                   mov    %eax,%eax
  400b3d:       48 39 45 f8             cmp    %rax,-0x8(%rbp)
  400b41:       73 34                   jae    400b77 <victim_function+0x4a>
    temp &= array2[array1[x] * 512];
  400b43:       48 8d 15 d6 c5 2c 00    lea    0x2cc5d6(%rip),%rdx        # 6cd120 <array1>
  400b4a:       48 8b 45 f8             mov    -0x8(%rbp),%rax
  400b4e:       48 01 d0                add    %rdx,%rax
  400b51:       0f b6 00                movzbl (%rax),%eax
  400b54:       0f b6 c0                movzbl %al,%eax
  400b57:       c1 e0 09                shl    $0x9,%eax
  400b5a:       48 63 d0                movslq %eax,%rdx
  400b5d:       48 8d 05 9c f6 2c 00    lea    0x2cf69c(%rip),%rax        # 6d0200 <array2>
  400b64:       0f b6 14 02             movzbl (%rdx,%rax,1),%edx
  400b68:       0f b6 05 91 e1 2c 00    movzbl 0x2ce191(%rip),%eax        # 6ced00 <temp>
  400b6f:       21 d0                   and    %edx,%eax
  400b71:       88 05 89 e1 2c 00       mov    %al,0x2ce189(%rip)        # 6ced00 <temp>
  }
}
  400b77:       90                      nop
  400b78:       5d                      pop    %rbp
  400b79:       c3                      retq

However, clang generates the following code.

void victim_function(size_t x) {
  400ac0:       55                      push   %rbp
  400ac1:       48 89 e5                mov    %rsp,%rbp
  400ac4:       48 89 7d f8             mov    %rdi,-0x8(%rbp)
  if (x < array1_size) {
  400ac8:       48 8b 7d f8             mov    -0x8(%rbp),%rdi
  400acc:       8b 04 25 90 c0 6c 00    mov    0x6cc090,%eax
  400ad3:       89 c1                   mov    %eax,%ecx
  400ad5:       48 39 cf                cmp    %rcx,%rdi
  400ad8:       0f 83 2f 00 00 00       jae    400b0d <victim_function+0x4d>
    temp &= array2[array1[x] * 512];
  400ade:       48 8b 45 f8             mov    -0x8(%rbp),%rax
  400ae2:       0f b6 0c 05 a0 c0 6c    movzbl 0x6cc0a0(,%rax,1),%ecx
  400ae9:       00
  400aea:       c1 e1 09                shl    $0x9,%ecx
  400aed:       48 63 c1                movslq %ecx,%rax
  400af0:       0f b6 0c 05 40 f2 6c    movzbl 0x6cf240(,%rax,1),%ecx
  400af7:       00
  400af8:       0f b6 14 25 50 dc 6c    movzbl 0x6cdc50,%edx
  400aff:       00
  400b00:       21 ca                   and    %ecx,%edx
  400b02:       40 88 d6                mov    %dl,%sil
  400b05:       40 88 34 25 50 dc 6c    mov    %sil,0x6cdc50
  400b0c:       00
  }
}
  400b0d:       5d                      pop    %rbp
  400b0e:       c3                      retq
  400b0f:       90                      nop

Interestingly, the clang-compiled spectre binary is not able to read the secret data! (At least not in gem5. It is able to read the secret data on my native machine.)

We can look into the two traces to see the difference between the clang version and the GCC version.

The main difference is that in the clang version, the load generated by the instruction at 0x400af0 never completes (and thus, must not have been issued to the memory system).

I'm not sure the exact cause of this difference. It could be that the instruction uses a different addressing mode (movzbl 0x6cf240(,%rax,1),%ecx in clang vs movzbl (%rdx,%rax,1),%edx in GCC). If you have ideas, please leave a comment!

Either way, minor differences in the code generated can have large impacts on the speculative execution!

Effects of branch predictor

When I was first playing around with Spectre and gem5, I ran into a problem where I could only sometimes get Spectre to "work" with the out of order CPU. After significant digging, I found that the branch predictor chosen makes a big difference to how quickly the vulnerability happens. The trace below (with the same code as GCC 4.8 above) shows what happens when using the tournament branch predictor.

Here, we see that the original branch misprediction comes much earlier than the jump instruction in victim_function that is at address 0x401072. Thus, by the time the load instructions in victim_function are executed, the ROB and load-store queue resources have been taken by other instructions and the rogue loads are not issued to memory. There are still a few times that the two loads are executed speculatively, but it is much more rare than with the TAGE predictor. When using the TAGE branch predictor, only the exact branch that the attacker wants to mispredict is mispredicted.

This interestingly shows that a "smarter" system is actually more vulnerable to speculation-based attacks!

Inferring Kaveri's Shared Virtual Memory Implementation

2017-07-03T00:00:00-05:00

Note: Most of this work was completed in 2014. These results may not hold with current drivers and more recent hardware (e.g., Carrizo)

Kaveri is AMD's first APU (accelerated processing unit, e.g. a chip with both a CPU and GPU on-die) to provide an implementation of heterogeneous system architecture (HSA). HSA is set of standards from the HSA Foundation which provides transparent support for multiple compute platforms. The goal is to allow programmers to use the GPU to accelerate parts of their applications with minimal overhead, both in terms of performance overhead and programmer effort.

The key features which differentiate HSA from other similar technologies are

Shared virtual memory which allows the CPU and GPU to access the same data transparently.
Heterogeneous queuing which allows the user-mode program to offload work to the GPU without OS intervention. Also, it allows CPU to schedule work for the GPU, the GPU to schedule work for the CPU, the GPU to schedule work for itself, etc.
Architected signals and synchronization which is mostly used for implementing the heterogeneous queues, but also allows for complex asynchronous communication between different compute platforms.

There are many questions about how AMD has implemented these features in their first iteration of HSA hardware. In this post, I am going to focus on the implementation of shared virtual memory.

Shared Virtual Memory

Shared virtual memory allows the GPU to access data in exactly the same fashion as the CPU. Another way to put it, which is often used by the AMD folks, is shared virtual memory allows "pointer-is-a-pointer" semantics. For the programmer, this means that she can allocate memory when executing on the CPU (e.g. by using malloc, tmalloc, or a user-defined allocator) and access that data from the GPU without any special translations. There is no need to copy the data to the GPU, or to declare that the data may be accessed by the GPU in the future. To implement shared virtual memory there are two main hardware requirements:

The CPU and GPU caches must be kept coherent. This doesn't have to be traditional hardware coherence. A valid implementation would be to flush the CPU/GPU caches whenever a synchronization instruction is executed.
The GPU must access the CPU page tables (e.g. x86-64). This means that GPU needs either a dedicated memory management unit (MMU) or that its memory accesses must pass through the IOMMU. (Possibly another post to come on this later.)

Shared virtual memory, as described here, is similar to "universal virutal addressing" from NVIDIA. The key difference is that with shared virtual memory, the GPU is accessing the CPU page tables transparently. There is no need to declare that a pointer will be used by the GPU before the GPU accesses it.

Deducing Cache Parameters

There is a great paper from the early 90's which looks at several machines available at the time and determines their cache parameters. Apparently, back then, we didn't have resources like Intel's architecture pages which gives us at least some information on what is in a manufactured chip. The paper by Saavedra et. al (available here) presents a simple microbenchmark (described below) and formulas which take the runtime of the benchmark and tell you the cache parameters.

This table is an overview of my findings. Below, I detail the methods and data that lead to these results. Information in italics I'm less sure about than the rest of the data.

	Capacity	Miss Latency	Associativity
L1 Cache	16 KB (Per CU)	225 ns	64-way
L2 Cache	256 KB (8 32 KB banks)	50 ns	at least 64-way
L1 TLB	64MB-128MB reach 64-128 2MB entries 2046 32 KB or 64 2 MB entries	450 ns	at least 64-way
L2 TLB (none if 2MB pages)	512 MB reach 256 2 MB entries	220 ns	256-way (for 4K pages)

The microbenchmark

(note: the code used to generate the data in this notebook and the data itself can be found on github.)

The microbenchmark that I used is quite simple. It allocates a chunk of memory, then issues memory requests to that memory. By varying the amount of memory allocated and the stride between each memory access, you can control the cache's effect. For instance, if the size of the allocated memory is less than or equal to the cache size, we would expect the entire array to fit in the cache and every access to hit in the cache, experiencing only the cache hit latency. Alternatively, if the allocated memory was larger than the cache, and we used a stride equal to the cache line size, we would expect every access to miss in the cache, experiencing the cache miss latency on every access.

Although the benchmark is simple, things are complicated by prefetching, multi-level caches, TLBs, replacement policies, etc. Some of these are relatively easy to overcome (we can trick the prefetcher by pointer chasing, for instance). Other things, like replacement policies other than LRU, cause the data to be more noisy or to not follow the model laid out by Saavedra et. al exactly.

In the data below, I've tried to explain as much as I can about why we see the data, but there are still some holes in my understanding.

The data!

For each of the graphs below, I've run the microbenchmark with input arrays from 4 KB up to 4 GB and strides from 4 bytes to the size of the array.

Baseline data

This first set of data uses the GPU to initialize the data (i.e. the data is touched by the GPU before the test is performed). It also uses the default Linux allocator, which uses transparent huge pages (THP) in this case.

data = pd.read_csv(urlbase+'data-thp-gpu.csv', usecols=(4,5,6), skipinitialspace=True)
figure(figsize=(12,6.66))
title('Init on GPU with THP')
for i in range(12,33):
    plt(2**i, data=data)
legend(loc='upper right')
xscale('log', basex=2)
grid(b=True, which='major')
xlabel('stride')
ylabel('time (ns)')
annotTHP()
tight_layout()

Starting on the left (and with smaller inputs):

The first array size which experiences L1 misses (the first teal line) is 2¹⁵
- This implies that the L1 cache capacity is < 32 KB, likely 16 KB.
- Since the latency changes from 300ns to 525ns, we can also infer that the L1 miss latency is 225ns. This may seem like a very high miss latency; however, it's not out of the realm of possibility for GPU caches. I've found similar numbers for both NVIDIA and AMD discrete GPUs.
- Also, the stride at which the latency is a maximum for the 2¹⁵ line is 64 bytes, which implies a 64 byte line. The reason we know it's the line size is that until we stride at the line size or above, the cache misses are partially amortized by hitting multiple times within the same cache block.
Next, the next array size (2¹⁶) seems to have a significantly different latency for each access.
- I believe this is the L2 miss latency, which is much smaller than the L1 miss latency at only 50ns. I assume that this L2 miss latency is actually the DRAM latency. If this is the case, the 50ns is quite reasonable. In discrete GPUs, the DRAM latency is much higher, but 50ns is reasonable for DDR3.
- However, I do not believe that the L2 cache is only 32 KB. It wouldn't make much sense to include if it was so small. What I believe is going on here (and I've yet to develop a test to confirm this hypothesis) is that the L2 has a 512 byte "line size", or as it's likely implemented, 8 banks and the banks are addressed by simple address slicing. If this is true, then the L2 is actually 256 KB, a much more reasonable size.
Nothing else interesting happens until we get to arrays larger than 2²⁷. At this point, we start missing in another level of cache. It's likely that this "cache" is actually the TLB.
- Since there is an increase in latency for 2²⁷ and it doesn't level off until 2²⁸, it is likely the TLB reach is between these two sizes (64 MB and 128 MB).
- Looking at the 2²⁸ line, and the ones after it, we see that latency levels off after a stride of 2²¹ or 2 MB. Interestingly, this is the large page size for x86-64. Thus, it seems that the GPU is using huge pages... cool!
- If we use this page size of 2 MB, then the 64 MB to 128 MB reach implies that the TLB has 32 to 64 entries. This is quite reasonable for a modern TLB. An interesting question is whether this TLB is local to the compute unit, or if it's shared between many compute units on the GPU. This could be the local TLB, or it could be the IOMMU TLB.
- Finally, we can see that the TLB miss latency is quite large, almost 500ns. Again, this is quite reasonable, especially if we assume that a TLB miss will take 1-4 memory accesses. (Well, 1-3 in this case since the GPU is using 2 MB pages).

GPU data without THP

Next we are going to look at what happens with the same test if we disable THP. This forces the GPU to use 4 KB pages. Though, interestingly, looking at the data, the GPU uses 32 KB pages even though the underlying page size is 4 KB!

Let's look at the data. It's generated using exactly the same methods as before.

data = pd.read_csv(urlbase+'data-nothp-gpu.csv', usecols=(4,5,6), skipinitialspace=True)
figure(figsize=(12,6.66))
title('Init on GPU without THP')
for i in range(12,33):
    plt(2**i, data=data)
legend(loc='upper right')
xscale('log', basex=2)
grid(b=True, which='major')
xlabel('stride')
ylabel('time (ns)')
annotNoTHP()
tight_layout()

Most of this data is the same as when using THP. The main difference is that we can see TLB misses with a much small array than in the previous data and that there are two TLB "line sizes".

Looking at the first level TLB, we see the line size is 32 KB. This actually isn't that strange of a "page" size. It's 8 4 KB pages, which happens to be the number of page translations that can fit in one 64 byte cache line. My hypothesis is that the GPU is using something like a sub-blocked TLB.
Interestingly, this TLB seems huge (2048 32 KB entries), especially if we compare it to the TLB in the first data. Since it's so big, my guess is that it's at the IOMMU and shared by the whole GPU, not a per-CU TLB. Additionally, there may be more sub-blocking going on all the way up to the 2 MB page size. Sub-blocking up to 2 MB pages lines up with the previous THP data the best. If the used 2 MB super-blocks then it would have 64 entries, which lines up with the L1 TLB in the previous data (which we found to be 64-128 entries).
Next, we again see the 2 MB page, but this time only for arrays larger than 512 MB. I find this quite strange that we didn't see this level of TLB in the previous data that had only 2 MB pages. It seems that this "L2 TLB", whatever it is, has an additional miss penalty not seen in the previous data. Assuming this TLB uses 2 MB pages, then it would have 256 entries.

I'm honestly not too sure what exactly is going on here. The data is pretty clear that there's two levels of TLB, but I'm not sure how these TLB levels relate to the TLB in the first set of data with 2 MB pages.

CPU first touch

Next, we're going to look at what happens when the GPU doesn't touch the data before the test is performed. For this test, I used the CPU to initialize the data and then performed the test on the GPU. Below we have the data both when THP is enabled and disabled. Since there aren't too many new data points in these two graphs I'll talk about them together below the graphs.

data = pd.read_csv(urlbase+'data-thp-cpu.csv', usecols=(4,5,6), skipinitialspace=True)
figure(figsize=(12,6.66))
title('Init on CPU with THP')
for i in range(12,33):
    plt(2**i, data=data)
legend(loc='upper right')
xscale('log', basex=2)
grid(b=True, which='major')
xlabel('stride')
ylabel('time (ns)')
annotTHP()
tight_layout()

data = pd.read_csv(urlbase+'data-nothp-cpu.csv', usecols=(4,5,6), skipinitialspace=True)
figure(figsize=(12,6.66))
title('Init on CPU without THP')
for i in range(12,33):
    plt(2**i, data=data)
legend(loc='upper right')
xscale('log', basex=2)
grid(b=True, which='major')
xlabel('stride')
ylabel('time (ns)')
annotNoTHP()
tight_layout()

The key difference between these two graphs and the first two (where the data was first initialized on the GPU) is for arrays of sizes 512 MB and above we see some pretty strange data. The way I interpret this data, and I may be totally wrong about this, is that the GPU caches are disabled. I believe this is true because of the latency seen using the word (4 byte) stride. Even in this case, when you would expect to get intra-block hits, the latency equal to the cache miss latency.

Although disabling the GPU caches really hurts performance for this microbenchmark, it may make sense for real applications. Maybe the runtime is somehow detecting that the data is written by the CPU and assumes that if the GPU wants up to date data that the highest performance option is to disable the caches (rather than rely on cache coherence).

I have considered other possibilities, like 1 GB pages, page faults, etc. But I can't come up with any other reason other than the above for this data.

Conclusions

There are still many unanswered questions about the design of the caches in Kaveri. This post just scratches the surface. I think the most interesting things that I learned are some of the details of the GPU MMU on Kaveri. (This may be most interesting to me since it's in my domain ;)). Also, it seems that accessing large arrays on the GPU that are first touched by the CPU results in very low performance. Since initializing data on the CPU and then processing that data on the GPU is a common idiom today, programmers will have to be careful to not use large arrays if they want to get high performance with AMD's current architecture.

Below I've included all of the other code that's required to get things to work. Questions and comments can be directed to me (Jason Lowe-Power) at powerjg@cs.wisc.edu. I'd like to thank my advisers Mark Hill and David Wood for helpful discussions about this data.

Code for generating the above graphs:

def plt(size, data):
    d = data[data['size'] == size]
    plot(d['stride'], d['overall_kernel_time']/100/1024*1e9, label='2^%d bytes' % log2(size))

def annotNoTHP():
    # L1
    hlines(525, 2**3, 2**5, lw=3)
    hlines(300, 2**3, 2**5, lw=3)
    vlines(2**4, 300, 525, lw=2)
    text(2**3, 350, "L1 miss latency\n~225 ns", rotation=90, va='bottom')
    vlines(2**6, 200, 525, lw=2, linestyle='--')
    text(2**6+10, 250, "Line size: 64 bytes")
    annotate("L1 capacity: 16 KB", xy=(2**9-100,350), xytext=(2**5+2**4-5, 400),
             arrowprops=dict(arrowstyle="->",
                           connectionstyle="arc3,rad=.2"))

    # L2
    hlines(570, 2**6, 2**8, lw=3)
    hlines(530, 2**6, 2**8, lw=3)
    vlines(2**7, 530, 570, lw=2)
    text(2**7, 580, "L2 miss latency\n~50 ns", ha='center')
    vlines(2**9, 200, 570, lw=2, linestyle='--')
    text(2**9+100, 210, "L2 \"Line size\": 512 bytes => 8 banks")
    annotate("L2 capacity: 256 KB\n(32 KB * 8 banks)", xy=(2**9+2**8,450), xytext=(2**9, 650),
             arrowprops=dict(arrowstyle="->",
                           connectionstyle="arc3,rad=.2"))

    # L1 TLB
    hlines(980, 2**17, 2**19, lw=3)
    hlines(570, 2**17, 2**19, lw=3)
    vlines(2**18, 570, 980, lw=2)
    text(2**16, 700, "L1 TLB miss latency\n~450 ns", ha='center')
    vlines(2**15, 200, 980, lw=2, linestyle='--')
    text(2**15+2**14, 250, "L1 TLB page size: 32 KB (8 4KB pages)")
    annotate("L1 TLB reach: 64 MB\n(2048 entries)", xy=(2**20+2**19,650), xytext=(2**10, 900),
             arrowprops=dict(arrowstyle="->",
                           connectionstyle="arc3,rad=.2"))

    # L2 TLB
    hlines(1200, 2**19, 2**21, lw=3)
    hlines(980, 2**19, 2**21, lw=3)
    vlines(2**20, 980, 1200, lw=2)
    text(2**20, 1250, "L2 TLB miss latency\n~220 ns", ha='center')
    vlines(2**21, 200, 1200, lw=2, linestyle='--')
    text(2**21+2**19, 210, "L2 TLB page size: 2 MB")
    annotate("L2 TLB reach: 512 MB\n(256 entries)", xy=(2**23+2**22,700), xytext=(2**13, 1100),
             arrowprops=dict(arrowstyle="->",
                           connectionstyle="arc3,rad=.2"))

def annotTHP():
    # L1
    hlines(525, 2**3, 2**5, lw=3)
    hlines(300, 2**3, 2**5, lw=3)
    vlines(2**4, 300, 525, lw=2)
    text(2**3, 350, "L1 miss latency\n~225 ns", rotation=90, va='bottom')
    vlines(2**6, 200, 525, lw=2, linestyle='--')
    text(2**6+10, 250, "Line size: 64 bytes")
    annotate("L1 capacity: 16 KB", xy=(2**9-100,350), xytext=(2**5+2**4-5, 400),
             arrowprops=dict(arrowstyle="->",
                           connectionstyle="arc3,rad=.2"))

    # L2
    hlines(570, 2**6, 2**8, lw=3)
    hlines(530, 2**6, 2**8, lw=3)
    vlines(2**7, 530, 570, lw=2)
    text(2**7, 580, "L2 miss latency\n~50 ns", ha='center')
    vlines(2**9, 200, 570, lw=2, linestyle='--')
    text(2**9+100, 210, "L2 \"Line size\": 512 bytes => 8 banks")
    annotate("L2 capacity: 256 KB\n(32 KB * 8 banks)", xy=(2**9+2**8,450), xytext=(2**9, 650),
             arrowprops=dict(arrowstyle="->",
                           connectionstyle="arc3,rad=.2"))

    # TLB
    hlines(980, 2**17, 2**19, lw=3)
    hlines(570, 2**17, 2**19, lw=3)
    vlines(2**18, 570, 980, lw=2)
    text(2**16, 700, "TLB miss latency\n~450 ns", ha='center')
    vlines(2**21, 200, 980, lw=2, linestyle='--')
    text(2**21+2**19, 250, "TLB page size: 2 MB")
    annotate("TLB reach: 64-128 MB\n(32-64 entries)", xy=(2**21,700), xytext=(2**13, 900),
             arrowprops=dict(arrowstyle="->",
                           connectionstyle="arc3,rad=.2"))
urlbase = "https://raw.githubusercontent.com/powerjg/hsa-micro-benchmarks/master/caches/data/A10-7850K/"
import pandas as pd

Setting up gem5 full system

2017-01-13T00:00:00-06:00

This is partially a followup to Creating disk images for gem5 and partially how to setup x86 full system for gem5. In this post, I'll discuss how to create a disk image from scratch and start using it with gem5.

It is important for computer architecture research to use the most up-to-date software on the systems we are simulating. Too much computer architecture research reports results using kernels from 5+ years ago or ancient system software Hopefully, this post will help others be able to keep up with the ever-changing system software. This way, researchers can use up-to-date versions of Linux and easily update their kernels.

This post takes a different approach than Creating disk images for gem5. Instead of using the gem5 tools, this post uses qemu to create, edit, and set up the disk for gem5 usage.

This post assumes that you have installed qemu on your system. In Ubuntu, this can be done with

sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils

I also assume you have downloaded and built gem5. All of the full system examples use the simple full system scripts that are covered in Learning gem5.

Step 1: Create an empty disk

Using the qemu disk tools, create a blank raw disk image. In this case, I chose to create a disk named "ubuntu-test.img" that is 8GB.

qemu-img create ubuntu-test.img 8G

Step 2: Install ubuntu with qemu

Now that we have a blank disk, we are going to use qemu to install Ubuntu on the disk. I would encourage you to use the server version of Ubuntu since gem5 does not have great support for displays. Thus, the desktop environment isn't very useful.

First, you need to download the installation CD image from the Ubuntu website.

Next, use qemu to boot off of the CD image, and set the disk in the system to be the blank disk you created above. Ubuntu needs at least 1GB of memory to install correctly, so be sure to configure qemu to use at least 1GB memory.

qemu-system-x86_64 -hda ../gem5-fs-testing/ubuntu-test.img -cdrom ubuntu-16.04.1-server-amd64.iso -m 1024 -enable-kvm -boot d

With this, you can simply follow the on-screen directions to install Ubuntu to the disk image. The only gotcha in the installation is that gem5's IDE drivers don't seem to play nicely with logical paritions. Thus, during the Ubuntu install, be sure to manually partition the disk and remove any logical partitions. You don't need any swap space on the disk anyway, unless you're doing something specifically with swap space.

Step 3: Boot up and install needed software

Once you have installed Ubuntu on the disk, quit qemu and remove the -boot d option so that you are not booting off of the CD anymore. Now, you can again boot off of the main disk image you have installed Ubuntu on.

Since we're using qemu, you should have a network connection (although ping won't work). When booting in qemu, you can just use sudo apt-get install and install any software you need on your disk.

qemu-system-x86_64 -hda ../gem5-fs-testing/ubuntu-test.img -cdrom ubuntu-16.04.1-server-amd64.iso -m 1024 -enable-kvm

Step 4: Build a kernel

Next, you need to build a Linux kernel. Unfortunately, the out-of-the-box Ubuntu kernel doesn't play well with gem5. See the error below.

First, you need to download latest kernel from kernel.org. Then, to build the kernel, you are going to want to start with a known-good config file.The config file that I'm used for kernel version 4.8.13 can be downloaded here. Then, you need to move the good config to .config and the run make oldconfig which starts the kernel configuration process with an existing config file.

At this point you can select any extra drivers you want to build into the kernel. Note: You cannot use any kernel modules unless you are planning on copying the modules onto the guest disk at the correct location. All drivers must be built into the kernel binary.

It may be possible to use modules by compiling the binary on the guest disk via qemu, but I have not tested this.

Finally, you need to build the kernel.

Step 5: Update init script

By default, gem5 expects a modified init script which loads a script off of the host to execute in the guest. To use this feature, you need to follow the steps below.

Alternatively, you can install the precompiled binaries for x86 found on my website: From qemu, you can run the following, which completes the above steps for you.

wget http://cs.wisc.edu/~powerjg/files/gem5-guest-tools-x86.tgz
tar xzvf gem5-guest-tools-x86.tgz
cd gem5-guest-tools/
sudo ./install

Now, you can use the system.readfile parameter in your Python config scripts. This file will automatically be loaded (by the gem5init script) and executed.

Manually installing the gem5 init script

First, build the m5 binary on the host.

cd util/m5
make -f Makefile.x86

Then, copy this binary to the guest and put it in /sbin. Also, create a link from /sbin/gem5.

Then, to get the init script to execute when gem5 boots, create file /lib/systemd/system/gem5.service with the following:

[Unit]
Description=gem5 init script
Documentation=http://gem5.org
After=getty.target

[Service]
Type=idle
ExecStart=/sbin/gem5init
StandardOutput=tty
StandardInput=tty-force
StandardError=tty

[Install]
WantedBy=default.target

Enable the gem5 service and disable the ttyS0 service.

systemctl enable gem5.service

Finally, create the init script that is executed by the service. In /sbin/gem5init:

#!/bin/bash -

CPU=`cat /proc/cpuinfo | grep vendor_id | head -n 1 | cut -d ' ' -f2-`
echo "Got CPU type: $CPU"

if [ "$CPU" != "M5 Simulator" ];
then
    echo "Not in gem5. Not loading script"
    exit 0
fi

# Try to read in the script from the host system
/sbin/m5 readfile > /tmp/script
chmod 755 /tmp/script
if [ -s /tmp/script ]
then
    # If there is a script, execute the script and then exit the simulation
    su root -c '/tmp/script' # gives script full privileges as root user in multi-user mode
    sync
    sleep 10
    /sbin/m5 exit
fi
echo "No script found"

Problems and (some) solutions

Failed to early mount API filesystems

Write protecting the kernel read-only data: 8192k
Freeing unused kernel memory: 1956K (ffff880001417000 - ffff880001600000)
Freeing unused kernel memory: 456K (ffff88000178e000 - ffff880001800000)
[!!!!!!] Failed to early mount API filesystems, freezing.

Solutions tried: Enable cgroups in the kernel. I think. Nope! I think this is the same as the problem below mount-problem.

Can't mount /dev

Failed to mount devtmpfs at /dev: No such device
Freezing execution.

Something like the above (this was taken from arch linux boot). The problem is that that the right devfs is not compiled into the kernel. You need to make sure that devtmpfs is enabled.

panic: KVM: Unexpected exit (exit_reason: 8)

Exit reason 8 is "shutdown". See http://lxr.free-electrons.com/source/include/uapi/linux/kvm.h#L188. This seems to happen when there is a triple fault: http://lxr.free-electrons.com/source/arch/x86/kvm/x86.c#L6498

I get this error every time I try to boot the unmodified Ubuntu kernel. I don't know how to solve this problem. Instead of trying to solve the problem, I used a different config file for "oldconfig" when I compiled the kernel from scratch.

Slow boot

[ TIME ] Timed out waiting for device dev-di...\x2da115\x2de3f263d7b53a.device.
[DEPEND] Dependency failed for /dev/disk/by-...382-f41d-4c99-a115-e3f263d7b53a.
[DEPEND] Dependency failed for Swap.

This may happen if you have changed the disk without updating the fstab on the disk. To fix it, you can boot the disk in qemu and update fstab with the correct UUID.

I ran into this when I was resizing the disk.

Disk is too small for what you want to do

Resizing an iso is pretty easy. You can use the same method you would if you wanted to resize a partition on a regular hard drive.

First, you need to resize the iso with qemu-image:

qemu-img resize ubuntu-test.img +8G

Now, you have a disk that has 8 GB of free space at the end of the disk. You need to resize the partitions to use this free space. To do this, I suggest using gparted just like you would for a real hard drive.

You can download a gparted ISO from http://gparted.org/livecd.php. Once you download the ISO, you can boot it with qemu the same way as we booted the installation CD. Then, once its booted you can select the disk you want to modify and follow the howto (http://gparted.org/display-doc.php%3Fname%3Dhelp-manual).

When to use 3D die-stacking for bandwidth-constrained big data workloads

2016-04-02T00:00:00-05:00

3D die-stacking is an exciting emerging technology. It promises extremely high bandwidth interconnects for relatively power (at least when compared to current memory technology). Some of my previous work has looked at how this increased bandwidth can significantly improve performance. This is especially true for bandwidth-constrained workloads, like database scan. However, my previous work has totally ignored the practicality of building these systems. In this post (and in the paper I am presenting at BPOE), I'd like to dig into whether, and/or when, 3D die-stacking is really appropriate, with a focus on big-data workloads.

While 3D die-stacking promises higher memory bandwidth, it comes at the cost of tiny capacities. Today, each stack of DRAM has 2–4 GB of capacity. Looking forward this should increase to 8–16 GB in the next few years due to increased DRAM densities and more chips per stack. Compare this to commodity DIMMs for which you can easily buy a single DIMM with 32 GB, and companies have announced DIMMs up to 128 GB.

One of the first observations I made when I began looking at 3D stacking is that current systems are incredibly unbalanced for high-bandwidth workloads like scan. There is currently a big push for in-memory databases, OEMs are designing systems for this workload. For instance, you can buy a system from Dell (the PowerEdge R930) with 3 TB of DRAM! Granted, it's likely to cost between $50,000 and $100,000, but Dell is happy to sell it to you. Oracle and IBM are also selling similar systems with even more DRAM per socket. So, looking at this system I asked, "What's the most complex query I can make if I only have 10ms to complete it?" I chose 10 ms as a proxy for a very aggressive SLA (service level agreement). The visualization below will give you an idea of what this looks like for the Dell system described above.

A visualization of the traditional system capacity and bandwidth.

In this system, you can only access 0.13% of the total data in 10ms! In fact, it would take over 7 seconds to read all of the data. (You can take the bandwidth (408 GB/s) and multiply by the time (10 ms) to find how much data you can access in 10 ms (4 GB). Then, compare this to the total memory ( 3 TB) to get 0.13% of memory capacity accessible in 10 ms.)

Let's compare this to the die-stacked system pictured below.

A possible design for a single node of a die-stacked system.

A visualization of the die-stacked system capacity and bandwidth.

For the die-stacked system, you can access 32% of the data in 10 ms. This is over 200 times more than in the traditional system.

So, it seems that the die-stacked system is way better for bandwidth constrained workloads. And this is the same conclusion I had in previous work. However, this ignores the fact that the capacity of the die-stacked system is more than an order of magnitude smaller than conventional systems. Because of this small capacity, you would need many more die-stacked systems to store the same amount of data as a single traditional system. For instance, if you have a 16 TB workload, you would need only 5–6 of the Dell servers, but you would need over 2000 die-stacked systems.

The systems required for a 16 TB workload. While you only need 6 traditional systems, you need over 2000 of the die-stacked systems (which don't even fit in the image)!

Having so many die-stacked systems greatly increases the power required to run the workload. And this is the fundamental trade off that we investigated in our BPOE paper. In a sentence, we found that if you require very high performance (like a 10ms SLA), the die-stacked system is the most reasonable system to build. However, if you have a less strict SLA (100ms or more) or if you have strict power constraints, then the traditional system is the best.

To reach these conclusions, we developed a simple back-of-the-envelope style analytical model for the power, performance, and capacity of these systems. If you're interested in more details, you can find the entire model here: 3D Bandwidth Model. You can download the IPython (Jupyter) Notebook from that page and run the model interactively on your local computer. Of course, you can also read the paper. You can download it from here: Paper link.

Creating disk images for gem5

2015-11-24T00:00:00-06:00

When using gem5 in full-system mode, you have to have a disk image with the operating system and all of your data on it. This is just like having a physical disk in a physical machine. In this post, I'm going to walk through how to create a new disk and install a (semi-)current version of Ubuntu on the disk. By the end of this post, you should be able to create your own disk with whatever extra data and applications you want.

This post assumes that you have already checked out a version of gem5 and can build and run gem5 in full-system mode. The Learning gem5 documentation is a good place to start. This post uses the x86 ISA for gem5, and is mostly applicable to other ISAs. More details on setting up ARM systems can be found on the gem5 wiki: http://gem5.org/Ubuntu_Disk_Image_for_ARM_Full_System.

In the future, this post may be folded into Learning gem5.

Creating a blank disk image

The first step is to create a blank disk image (usually a .img file). Luckily, the gem5 developers have already made this easy with a tool that is simple to use. To create a blank disk image, which is formatted with ext2 by default, simply run the following.

> util/gem5img.py init ubuntu-14.04.img 4096

This command creates a new image, called "ubuntu-14.04.img" that is 4096 MB. This command may require you to enter the sudo password, if you don't have permission to create loopback devices. You should never run commands as the root user that you don't understand! You should look at the file util/gem5img.py and ensure that it isn't going to do anything malicious to your computer!

We will be using util/gem5img.py heavily throughout this post, so you may want to understand it better. If you just run util/gem5img.py, it displays all of the possible commands.

Usage: %s [command] <command arguments>
where [command] is one of
    init: Create an image with an empty file system.
    mount: Mount the first partition in the disk image.
    umount: Unmount the first partition in the disk image.
    new: File creation part of "init".
    partition: Partition part of "init".
    format: Formatting part of "init".
Watch for orphaned loopback devices and delete them with
losetup -d. Mounted images will belong to root, so you may need
to use sudo to modify their contents

Copying root files to the disk

Now that we have created a blank disk, we need to populate it with all of the OS files. Ubuntu distributes a set of files explicitly for this purpose. You can find the Ubuntu core distribution for 14.04 at http://cdimage.ubuntu.com/ubuntu-core/releases/14.04/release/ Since I am simulating an x86 machine, I chose the file ubuntu-core-14.04-core-amd64.tar.gz. Download whatever image is appropriate for the system you are simulating.

Next, we need to mount the blank disk and copy all of the files onto the disk.

mkdir mnt
../../util/gem5img.py mount ubuntu-14.04.img mnt
wget http://cdimage.ubuntu.com/ubuntu-core/releases/14.04/release/ubuntu-core-14.04-core-amd64.tar.gz
sudo tar xzvf ubuntu-core-14.04-core-amd64.tar.gz -C mnt

The next step is to copy a few required files from your working system onto the disk so we can chroot into the new disk. We need to copy /etc/resolv.conf onto the new disk.

sudo cp /etc/resolv.conf mnt/etc/

Setting up gem5-specific files

Create a serial terminal

By default, gem5 uses the serial port to allow communication from the host system to the simulated system. To use this, we need to create a serial tty. Since Ubuntu uses upstart to control the init process, we need to add a file to /etc/init which will initialize our terminal. Also, in this file, we will add some code to detect if there was a script passed to the simulated system. If there is a script, we will execute the script instead of creating a terminal.

Put the following code into a file called /etc/init/tty-gem5.conf

# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again, unless there is a script passed to gem5.
# If there is a script, the script is executed then simulation is stopped.

start on stopped rc RUNLEVEL=[12345]
stop on runlevel [!12345]

console owner
respawn
script
   # Create the serial tty if it doesn't already exist
   if [ ! -c /dev/ttyS0 ]
   then
      mknod /dev/ttyS0 -m 660 /dev/ttyS0 c 4 64
   fi

   # Try to read in the script from the host system
   /sbin/m5 readfile > /tmp/script
   chmod 755 /tmp/script
   if [ -s /tmp/script ]
   then
      # If there is a script, execute the script and then exit the simulation
      exec su root -c '/tmp/script' # gives script full privileges as root user in multi-user mode
      /sbin/m5 exit
   else
      # If there is no script, login the root user and drop to a console
      # Use m5term to connect to this console
      exec /sbin/getty --autologin root -8 38400 ttyS0
   fi
end script

Setup localhost

We also need to set up the localhost loopback device if we are going to use any applications that use it. To do this, we need to add the following to the /etc/hosts file.

127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Update fstab

Next, we need to create an entry in /etc/fstab for each partition we want to be able to access from the simulated system. Only one partition is absolutely required (/); however, you may want to add additional partitions, like a swap partition.

The following should appear in the file /etc/fstab.

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system>    <mount point>   <type>  <options>       <dump>  <pass>
/dev/hda1          /               ext3            noatime         0 1

Copy the `m5` binary to the disk

gem5 comes with an extra binary application that executes pseudo-instructions to allow the simulated system to interact with the host system. To build this binary, run make -f Makefile.<isa> in the gem5/m5 directory, where <isa> is the ISA that you are simulating (e.g., x86). After this, you should have an m5 binary file. Copy this file to /sbin on your newly created disk.

After updating the disk with all of the gem5-specific files, unless you are going on to add more applications or copying additional files, unmount the disk image.

> util/gem5img.py umount mnt

Install new applications

The easiest way to install new applications on to your disk, is to use chroot. This program logically changes the root directory ("/") to a different directory, mnt in this case. Before you can change the root, you first have to set up the special directories in your new root. To do this, we use mount -o bind.

> sudo /bin/mount -o bind /sys mnt/sys
> sudo /bin/mount -o bind /dev mnt/dev
> sudo /bin/mount -o bind /proc mnt/proc

After binding those directories, you can now chroot:

> sudo /usr/sbin/chroot mnt /bin/bash

At this point you will see a root prompt and you will be in the / directory of your new disk.

You should update your repository information.

> apt-get update

You may want to add the universe repositories to your list with the following commands. Note: The first command is require in 14.04.

> apt-get install software-properties-common
> add-apt-repository universe
> apt-get update

Now, you are able to install any applications you could install on a native Ubuntu machine via apt-get.

Remember, after you exit you need to unmount all of the directories we used bind on.

> sudo /bin/umount mnt/sys
> sudo /bin/umount mnt/proc
> sudo /bin/umount mnt/dev

gem5 Horrors and what we can do about it

2015-06-09T00:00:00-05:00

This post is a post which mostly follows the talk that I am giving at the gem5 Users Workshop. This post contains some more details on problems that I skipped in my talk and some references that I was not able to include in a presentation. You can view my presentation on Google Drive here.

I <3 gem5

Before I get into the negative aspects of gem5, I first want to point out that it is a great tool. gem5 is used by a large number of computer architecture researchers, both in industry and in academia. Here at Wisconsin, and at other universities, gem5 is used in the classroom to teach students about computer architecture and how to do computer architecture research.

gem5 is, without a doubt, the most full-featured architecture simulator. It leverages execute-at-execute semantics for high-fidelity cycle-by-cycle simulation. gem5 can boot a mostly unmodified Linux image. It has multiple different CPU and memory models. gem5 has a modular design which makes it simple to embed and extend. This has allowed gem5 to be used a large number of projects (see http://gem5.org/Projects and http://gem5.org/Publications).

However, as great as gem5 is, its growth has not been without pain. Now that gem5 is nearing 15 years of development (if you include the original m5 and GEMS project from which gem5 was born), I believe it's time to look at some of its deficiencies and talk about what we can do to mitigate them.

gem5 horrors

Below, I discuss a few specific pain points with that I and others have experienced with gem5. However, before I get to that, I'd like to talk about what I think the root of these issues are. gem5 has two main problems

There is no formal governance model.
The gem5 developers do not think of the user first.

Later, after I give some examples of specific problems, I will discuss what I think can be done to fix these to issues.

Next I discuss four specific "gem5 horrors" that either I have personally experienced or I have talked to others who have experienced them. These issues are deeper that just bugs, even if sometimes they can be solved with simple changes. After describing each issue, I will also quickly discuss a possible way to mitigate the problem.

Horror 1: Merges

There are a number of projects that build on top of gem5. In fact, I would argue that this is the main use case for gem5. Everyone that I know who uses gem5 for research, takes the mainline gem5 and builds their own changes on top of it.

The problem with this model, where people build on top of gem5, is that when new features are added or bugs are fixed in the mainline, downstream users have to consume these changes. If the downstream users do a good job managing their patch queues, this should be a straightforward thing to do. However, I have found that even when careful development practices are followed, merging gem5 changes is incredibly difficult.

Below I discuss a few specific problems that I have run into when merging new changes in gem5. I believe the problems can be summed up with two high-level issued we currently have in gem5.

There is no well-defined static API. The interface to different modules is constantly in a state of flux.
The regression suite we have in gem5 has poor coverage. There are many features that users depend on that are not covered by the regression tester.

Merge headache #1: Pointless code changes

Examples from Ruby and Slicc and packet.

Merge headache #2: Features break between versions

Ruby backing store, checkpointing

Merge headache #3: APIs are a moving target

Example with the minimal gem5 script.

How to mitigate

I believe that there are two things we can can do as the gem5 development community to make merging upstream changes much easier. First, we need a stable set of APIs. Second, we need a robust testing and regression structure. I discuss some specifics of these two characteristics below.

Stable APIs

Today in gem5, it is just as easy to change widely used interfaces, like the port interface, as it is to change the implementation of a rarely used function. We need to change this. I think that we need to choose a set of interfaces and make them stable. This is similar to how the Linux kernel operates.

Once we have chosen a set of stable interfaces, I'm not suggesting that they never change, only that it should be more onerous to change stable APIs than other things. Additionally, this has the added benefit that "gem5-stable" can actually mean something. We can now have a stable version, which has non-changing APIs, and a dev version that we can't necessarily count on to have constant APIs.

I personally do not know what the API should be. I would like to see the community come together and talk about what they see as important interfaces. Then, once we find these interfaces, we can architect these interfaces and hopefully make gem5 easier to use.

Testing structure

I do not think that this is a very controversial issue, but gem5 needs a better regression structure. If all of the features that we used in gem5-gpu had been part of the regression suite, then we would have had many less problems.

Again, I do not know exactly how to make the regression suite better, but I do think a good idea would be to require new features, and bug fixes, to include a unit-test or something like that. We really need a softeare engineer to sit down and architect a new regression system. This would be a great project for someone who is new to the gem5 codebase.

Horror 2: Configuration files

gem5 has an incredibly flexible configuration system. But with flexibility often come complexity. In fact, I ran SLOCcount on the configs directory and found there was more than 4000 lines of Python code. According to the SLOCcount tool, this means there was 16 person-months and a quarter of a million dollars worth of code here!

All of this complexity causes a number of issues. In my talk, I touched on the fact that the defaults are confusing, and in some cases inconsistent.

How to mitigate

Since the m5 and GEMS integration, I have noticed a trend that the number of command line parameters has continued to grow significantly. It seems that every time a new feature has been added, we have added some new command line parameters as well. I think this is the wrong way to do it.

There is an amazing C++-Python wrapper in gem5. We should be taking advantage of the scripting capabilities of Python.

I have created a simple script that is under 30 lines of Python. I think we need to encourage our users to script in Python instead of adding more and more command line parameters. Which, in my experience, really just leads to scripting in bash instead of in Python anyway.

Horror 3: Unexpected results

This was a very surprising error that I ran into while working on creating a homework assignment for a graduate-level computer architecture course. The point of the homework was to compare the performance of instruction latency versus instruction throughput. I wanted the students to take a particular instruction and change the number of execution units, the latency, and how much the units were pipelined. To do this, we looked at the divide instruction, since it is a long latency instruction. Below is the code that we used:

for (int i = 0; i < N; i++) {
      Y[i] = X[i] / alpha + Y[i];
}

In this code, every divide is totally independent from every other. Therefore, we would expect that with he out-of-order CPU, that if the divide is pipelined it the code will speedup by how much the divide unit is pipelined.

To test this, I looked at two different configurations, a 10 cycle latency divide with no pipelining, and a 10 cycle latency divide that is fully pipelined. Below is the data I found for ARM and x86. I only changed the "obvious" options. Each functional unit has an option for the execution latency and issue latency. If the issue latency is 1, then the functional unit is fully pipelined. (Now this is a boolean flag.) All of the data is relative to x86 with no pipelining.

Configuration	Latency	Issue lat.	x86 Perf	ARM Perf
No Pipeline	10 cycles	10 cycles	1.0x	8.0x
Full Pipeline	10 cycles	1 cycle	1.0x	9.6x (1.2x)

There are two very weird results in this data. First, when we fully pipelined the divide unit, there was no performance change (at all!!) in x86. Second, when running the exact same cod with ARM, there was a 8x speedup compared to x86! I find it very hard to believe that the ARM ISA is inherently better at divide than x86.

How to mitigate

This is a much harder problem to mitigate than the others on this list. Nilay Vaish has taken a step in the right directions with these two patches on reviewboard http://reviews.gem5.org/r/2744/ and http://reviews.gem5.org/r/2744/, which have been incorporated in gem5.

The underlying problem is that the implementation for ARM and x86 are totally distinct. It is not clear to me what the right way to unify the ISA implementation are. As a stop-gap, developers who are working on implementing x86 features, need to make sure that they perform similarly to ARM features. Maybe a solution is to have a single set of C programs which exercise all ISAs and compare the performance across ISAs. There should be some performance differences, but not an order of magnitude.

Horror 4: Lack of new-user support

How to mitigate

What I think we need to do is to create a "gem5 for Dummies" book or a "Learning gem5" book. This book would be similar to Learning Python or Learning Mercurial. The book would be open source for anyone to contribute to. In fact, it should be required to update the book if a developer makes an API-breaking change.

An initial implementation of this book, which currently only includes about a chapter of "getting started" and is in fact already out of date can be found here: gem5-tutorial. I began working on this in conjunction with the graduate computer architecture class at Wisconsin, so it may currently have some Wisconsin-specific text. I hope to continue working on this in my copious free time.

There are many other horrors that other people experience as well. Here I only discussed some of the horrors that I have heard people discussing. The purpose in presenting these horrors is not to say that gem5 is a bad simulator! The purpose is to highlight how there are currently issues that need to be addressed by the gem5 development community.

What can we do about it?

A lot of the problems that I have discussed above come down to poor software engineering. And yes, we are architects, not software engineers, and there are a lot of things we could do better if we just focused on software engineering. However, I do not think that this is the underlying issue.

I believe these four horror stem from two systemic problems in the gem5 development community.

There is no formal governance model.
The gem5 developers do not think of the user first.

I believe that if we start to solve these high-level issues, gem5 will be a much better tool for everyone. Next, I discuss one possible way to address these two points.

gem5 Foundation

First, I want to say that I do not believe this is the only way, or the right way, to move gem5 forward. This is one possibility that I believe will make gem5 a better tool. I hope that this is a place to begin the discussion and I am sure that others in our community can come up with even better suggestions that this!

I think we should create a gem5 Foundation. The gem5 Foundation will be the center for the gem5 community. It will be a formal way for the community to set goals and push gem5 forward.

There are two main things I think the gem5 Foundation can help us with. It can set up a formal governance structure and be a place for outside interests to contribute money towards making gem5 better for everyone.

Formalizing a governance structure

First, we need a governance structure. This is a document which defines how decisions are made in the community, what matters to the community, how to contribute to the community, etc.

There is a lot of documentation on how to write governance models and what they are. OSS-Watch is a great source for this. Here is a link to a definition of a governance model, which does a much better job that I can explaining it. http://oss-watch.ac.uk/resources/governancemodels Additionally, here is a link to an example governance model from an academic open-source project: http://www.taverna.org.uk/about/legal-stuff/taverna-governance-model/

Money, Money, Money

What I think the main solution to all of these problems is to pay software developers not computer architects! to solve some of these problems. Already, within ARM and AMD there are a number of people who get paid to work on gem5. However, these companies do not have gem5's best interests as their key focus. Their focus is what ARM and AMD find interesting.

So, I think that if we have something like the gem5 Foundation, these companies and academia, can donate money towards coding things that are good for the community as a whole. The gem5 Foundation can hire software engineers to work on the parts of gem5 that grad students and researchers do not want to do. If you look at other academic communities, they often hire non researchers to do the "grunt work". Overall, I think this is a good idea for computer architects too, and specifically for gem5.

I recognize that this may be a crazy idea. I would love to hear what others think. I am sure we will have some interesting discussion at the gem5 workshop, and hopefully I will write another post with what other people thought! Feel free to leave comments below.

Setting up a gem5 container

2015-04-22T00:00:00-05:00

Linux containers (LXC) are a relatively new feature in Linux. From what I gather, they allow you to create a sandboxed environment to run applications. Containers are somewhere between processes and a full virtual machine. They allow you to use and link to any libraries or applications you need, but they don't cost the same high overheads as full virtualization.

For gem5, I've been told by some people (mostly Mike Swift) that I should look at containers and Docker to see if it would be something useful for the gem5 community. So, in this post, I walk through what it took to set up a container for gem5.

<I WANT AN INSET OR SOMETHING> After I completed the post, I came back to this section to evaluate whether the work I put into creating a gem5 container was worth it.

In a word: "no".

I don't see the use case for the gem5 container that I just created. Maybe it's that I don't understand what containers are, or how to distribute them. The way I see it, I have just created a useful tool for myself, but it's not much use to the community as a whole. Next, I'm going to try to develop a gem5 Docker application. Maybe that will be more useful.

Install and configure lxc in Ubuntu

The first step is to install lxc, obviously. For the most part, I followed the documentation from the Linux Containers .

In Ubuntu it was as simple as:

> sudo apt-get install lxc

Next, I needed to set up user-mode unprivileged containers for my primary user (jason). First, I found my userid from /etc/subuid

jason:100000:65536

Then, I added my username as an allowed user of the virtual ethernet device by adding the following line to /etc/lxc/lxc-usernet

jason veth lxcbr0 10

Then, as specified in the getting started page, I created the ~/.config/lxc directory and copied /etc/lxc/default.conf to ~/.config/lxc/default.conf. Finally, I added the following lines to that file. Note that I copied the nubers from /etc/subuid.

lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536

Finally, I needed to add one more line to ~/.config/lxc/default.conf. From what I can tell, there is a bug with my version of Ubuntu and AppArmor, though the pages I found describing the issue were not very clear. I saw the following error when I initially tried to start my container with lxc-start -n my-container -F:

lxc-start: lsm/apparmor.c: apparmor_process_label_set: 169 If you really want to start this container, set
lxc-start: lsm/apparmor.c: apparmor_process_label_set: 170 lxc.aa_allow_incomplete = 1
lxc-start: lsm/apparmor.c: apparmor_process_label_set: 171 in your container configuration file

To solve this problem, I added the follwing to ~/.config/lxc/default.conf.

lxc.aa_allow_incomplete = 1

Now, I was ready to create a new container for gem5.

Creating a container

You can use the following command to create a container based on the images from linuxcontainers.org:

> lxc-create -t download -n gem5-container

You will be shown a number of distributions, versions, and architecture options. I chose Ubuntu, trusty, and amd64.

Distribution: ubuntu
Release: trusty
Architecture: amd64

Now, you can start and log into the container with:

> lxc-start -n gem5-container -d
> lxc-attach -n gem5-container

Installing gem5 dependencies

The next step is installing of the dependencies for gem5. The current dependencies can be found on the gem5 website.

You can execute the following apt-get command to get all of gem5's dependencies:

> apt-get install build-essential gdb python-dev scons swig zlib1g-dev m4 libprotobuf-dev python-protobuf protobuf-compiler libgoogle-perftools-dev
> apt-get --no-install-recommends install mercurial

Finishing up

Next, I created a gem5 user, with the username gem5 and password gem5. This will allow us to keep gem5 relatively sandboxed when building and running.

> useradd gem5
> passwd gem5

Before downloading gem5, let's checkpoint our current state so we can come back to it in the future for a clean gem5-free system. First, we have to halt the container by using the halt command

> halt

After this, we are dumped back out to the host-system's terminal. Now we can clone the container with the following command:

> lxc-clone gem5-container gem5-depends

And restart the container:

> lxc-start -n gem5-container -d
> lxc-attach -n gem5-container

Downloading and building gem5

Now, let's change to the gem5 user and install and build gem5. We'll put gem5 in /opt/gem5.

> mkdir /opt/gem5
> chmod a+rw gem5
> su gem5
> cd /opt

After creating the directory, we can clone gem5!

> hg clone clone http://repo.gem5.org/gem5 gem5
> cd gem5

When you list the directory you should see the following from the gem5 repository:

build_opts  configs  COPYING  ext  LICENSE  README  SConstruct        src  system  tests  util

Now, we can build gem5.

scons build/X86/gem5.opt

You will be asked to install the style hook the first time you build gem5. Just press enter and the style hook will automatically be added to you .hg/hgrc file.

You're missing the gem5 style hook, which automatically checks your code
against the gem5 style rules on hg commit and qrefresh commands.  This
script will now install the hook in your .hg/hgrc file.
Press enter to continue, or ctrl-c to abort:

Now we have a full, sandboxed, development environment for gem5! We can use this to build and make changes to gem5 without affecting the system we are running on.

First post!

2015-04-08T00:00:00-05:00

This is my first post!

So, I've decided that I am going to try to keep a blog of sorts.

In my work in computer architecture, there are a number of things that I've done that don't warrant writing a whole paper on them, but it seems like it may be useful to other in the community. I foresee this being the biggest use of this blog. However, I may get distracted with other things now and then.

Jason's Website

Visualizing Spectre with gem5

Running Spectre

Running Spectre in gem5

Visualizing the out of order pipeline

Understanding the O3 pipeline viewer

Digging deeper into Spectre

Effects of compilers

GCC 7.2

Effects of branch predictor

Inferring Kaveri's Shared Virtual Memory Implementation

Shared Virtual Memory

Deducing Cache Parameters

The microbenchmark

The data!

Baseline data

GPU data without THP

CPU first touch

Conclusions

Code for generating the above graphs:

Setting up gem5 full system

Step 1: Create an empty disk

Step 2: Install ubuntu with qemu

Step 3: Boot up and install needed software

Step 4: Build a kernel

Step 5: Update init script

Manually installing the gem5 init script

Problems and (some) solutions

Failed to early mount API filesystems

Can't mount /dev

panic: KVM: Unexpected exit (exit_reason: 8)

Slow boot

Disk is too small for what you want to do

When to use 3D die-stacking for bandwidth-constrained big data workloads

Creating disk images for gem5

Creating a blank disk image

Copying root files to the disk

Setting up gem5-specific files

Create a serial terminal

Setup localhost

Update fstab

Copy the m5 binary to the disk

Install new applications

gem5 Horrors and what we can do about it

I <3 gem5

gem5 horrors

Horror 1: Merges

Merge headache #1: Pointless code changes

Merge headache #2: Features break between versions

Merge headache #3: APIs are a moving target

How to mitigate

Stable APIs

Testing structure

Horror 2: Configuration files

How to mitigate

Horror 3: Unexpected results

How to mitigate

Horror 4: Lack of new-user support

How to mitigate

What can we do about it?

gem5 Foundation

Formalizing a governance structure

Money, Money, Money

Setting up a gem5 container

Install and configure lxc in Ubuntu

Creating a container

Installing gem5 dependencies

Finishing up

Downloading and building gem5

First post!

This is my first post!

Copy the `m5` binary to the disk