https://bakhi.github.io/devmem/

Skip links

  * Skip to primary navigation
  * Skip to content
  * Skip to footer

Heejin Park

  * Mobile GPU
  * Posts
  * Contact

Toggle menu

Heejin Park

Heejin Park

SWE @ Apple
PhD @ Purdue
Follow

  * Sunnyvale, CA
  * Email
  * GitHub
  * Linkedin
  * Google Scholar
  * CV

Directly Access Your Physical Memory (dev/mem)

2 minute read

On this page

  * 1 What is /dev/mem?
  * 2 How to use
      + 2.1 Requisites
      + 2.2 Let's code
  * 3 Problem: impact of cache

1 What is /dev/mem?

"/dev/mem" is a character device file, image of the main memory of
system. It allows to directly access any phys address.

2 How to use

2.1 Requisites

To use /dev/mem, your kernel must be configured with
"CONFIG_STRICT_DEVMEM=n", or it prevent access from even privileged
user.

  * "CONFIG_STRICT_DEVMEM=y", the default kernel configuration in
    general, disallows to access RAM area via /dev/mem or only allows
    first 1MB size of RAM

FYI: "CONFIG_IO_STRICT_DEVMEM=y", disallows to access register via
dev/mem

2.2 Let's code

To start with, open dev/mem device file and map the phys address we
are interested in. Note that the phys address should be page-aligned.
Depending on request, read or write value to the mapped address. To
avoid a code sequence optimization from compiler, use volatile when
you read or write.

int mem_fd = open("/dev/mem", O_RDWR | O_SYNC);
void *map_base = mmap(NULL,
                        PAGE_SIZE,
                        PROT_READ | PROT_WRITE,
                        MAP_SHARED,
                        mem_fd,
                        phys_addr);     // phys_addr should be page-aligned.    

void *virt_addr = (char *)map_base + offset_in_page;

if (is_read) {
    read_result = *(volatile uint64_t*)virt_addr;
} else {        // write
    *(volatile uint64_t*)virt_addr = write_value;
}

                  Simple example for using /dev/mem

Useful tool: busybox.

                              [busybox]
 Example of devmem utility from busybox: it tries to directly access
          MMIO region of Mali GPU and achieves its GPU ID.

3 Problem: impact of cache

If you directly use RAM, the memory might be cached by the CPU which
possibly incurs cache coherence problem.

  * When you write, the exact value could not be written to the
    memory yet but CPU cache.
  * We don't know when it will be flushed.

Scenario: I implemented kernel module that allocates pages by
alloc_pages (low-level page request mechanism). When user-space
application requested, the module allocates the page and passes its
phys address to the user-space. The application starts to write
something into the allocated page by using /dev/mem.

Observation: When writing some values via /dev/mem, it seems the phys
memory is corrupted after the write is done as follows.

                             [devmem_0]
              Memory is being corrupted as time goes on

Guess: It seems that phys memory where we try to write was cached
before. Then we write it directly using dev/mem which bypasses the
CPU cache and later, the cache is flushed which corrupts the data we
wrote. This may be affected by other cores as well when the device
has multiple cores.

Troubleshooting:

i) turn off all the other cores except for the main core (cpu0) which
seems work (no data corruption; no flush from other cores)

echo 0 | sudo tee /sys/devices/system/cpu/cpu1/online

ii) map the allocated phys memory as DMA memory, which prevents
cached access.

struct page *p = alloc_pages(gfp, pool->order);
dma_addr_t dma_addr = dma_map_page(dev, p, 0, (PAGE_SIZE << pool->order), DMA_BIDIRECTIONAL);

Note: Depending on the memory region the phys address you try to
access, the mapping is done with either "pgprot_noncached" or
"pgprot_writecombined". For instance, MMIO region is mapped as
noncached while normal memory region as writecombined. It also
depends on the your kernel arch, so look into how it is implemented
in your kernel source code, "driver/char/ mem.c".

Tags: dev/mem, phys mem access

Updated: December 10, 2020

Share on

Twitter Facebook LinkedIn Previous Next

You may also enjoy

DSTAT - Resource Monitoring

1 minute read

When you want to see the information or statistics of system such as
IO devices, CPU and network, dstat can be the one that is easy to
utilize. dstat is a mo...

TMUX - A Terminal Multiplexer

1 minute read

You may want to see multiple terminals in a single screen. For
instance, you want to edit the code while monitoring the system
utilization or kernel log from...

DLXOS Setup

less than 1 minute read

Setup DLXOS with Ubuntu 20.04

Mali Bifrost - Cache Clean

2 minute read

What Invokes Cache Clean?

  * Follow:
  * Feed

(c) 2022 Heejin Park. Powered by Jekyll & Minimal Mistakes.