EDIT: Is there a better stackexchange site I should put this on?
I am trying use the DDR4 connected to the FPGA side of a zynq ultrascale chip, specifically on Xilinx's devboard zcu102. I am using Vivado's MIG to generate the controller and connect to the HPM0 AXI port of the main processor. The memory is 0x2000_0000 long and mapped to physical address 0x4_0000_0000.
The problem I am having is when accessing memory unaligned. I believe it to be some misconfiguration in the MMU due to a missing detail in the device tree.
One thing I know is that this arm (A53) is supposed to be able to turn unaligned accesses into multiple access to the memory where necessary. My getting a bus error is telling me that the MMU is misconfigured in some way but I'm not sure how.
The device tree is too long to put here, so I will put what I think are the most important parts directly in here and the rest as a pastebin.
amba_pl@0 {
#address-cells = <0x2>;
#size-cells = <0x2>;
compatible = "simple-bus";
ranges;
ddr4@400000000 {
compatible = "xlnx,ddr4-2.2";
reg = <0x4 0x0 0x0 0x20000000>;
alignment = <0x10>;
};
PERIPHERAL@ff380000 {
compatible = "xlnx,PERIPHERAL-1.0";
reg = <0x0 0xff380000 0x0 0x80000>;
};
PERIPHERAL@ff990000 {
compatible = "xlnx,PERIPHERAL-1.0";
reg = <0x0 0xff990000 0x0 0x10000>;
};
};
aliases {
ethernet0 = "/amba/ethernet@ff0e0000";
i2c0 = "/amba/i2c@ff020000";
i2c1 = "/amba/i2c@ff030000";
serial0 = "/amba/serial@ff000000";
serial1 = "/amba/serial@ff010000";
spi0 = "/amba/spi@ff0f0000";
};
memory {
#address-cells = <0x2>;
#size-cells = <0x2>;
device_type = "memory";
alignment = <0x10>;
reg = <0x4 0x0 0x0 0x20000000>;
};
I tried setting the alignment on both /memory and the ddr controller on the /amba_pl, but it doesn't seem to make a difference. In fact, it seems to be ignoring the /memory section, as it still boots off of main system memory and not the memory on the fpga.
I am able to access memory using mmap and devmem as such:
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>
#include <stdint.h>
#include <math.h>
#define ARRAY_TYPE uint16_t
int main(int argc, char *argv[]) {
if (argc != 2){
printf("Wrong arg count\n");
return -1;
}
volatile ARRAY_TYPE * ps = malloc(sizeof(ARRAY_TYPE)*10);
if (NULL==ps){
printf("Failed to malloc for ps\n");
return -1;
}
int fd = open("/dev/mem", O_RDWR | O_SYNC );
volatile ARRAY_TYPE *pl = mmap((void*)0x400000000, 0x20000000, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0x400000000);
if (pl == MAP_FAILED) {
perror("Can't map memory for pl");
printf("FAIL\n");
return -1;
}
//force them out of alignment by argv[1] bytes.
volatile ARRAY_TYPE *ps_unaligned = (volatile ARRAY_TYPE *)(((uint8_t *)ps) + atoi(argv[1]));
volatile ARRAY_TYPE *pl_unaligned = (volatile ARRAY_TYPE *)(((uint8_t *)pl) + atoi(argv[1]));
printf("trying ps with offset %d\n", atoi(argv[1]));
ps_unaligned[0] = 1;
printf("trying pl with offset %d\n", atoi(argv[1]));
pl_unaligned[0] = 1;
munmap(pl, 0x20000000);
free(ps);
close(fd);
return 0;
}
Whenever I try to access memory out of alignment on the main system memory it works fine. However when I access the memory hanging off the FPGA unaligned I get a bus error.
root@zynq: ~
17:22:08 $ ./memtest 0
trying ps with offset 0
trying pl with offset 0
root@zynq: ~
17:22:12 $ ./memtest 1
trying ps with offset 1
trying pl with offset 1
Bus error
root@zynq: ~
17:22:13 $ ./memtest 2
trying ps with offset 2
trying pl with offset 2
root@zynq: ~
17:22:14 $ ./memtest 3
trying ps with offset 3
trying pl with offset 3
Bus error
I've also looked at the virtual memory mappings of the memory region by looking at /proc/$pid/smaps during a gdb session: Here's the section containing the system memory allocated with devmem/mmap:
400000000-420000000 rw-s 400000000 00:06 1054 /dev/mem
Size: 524288 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 0 kB
Pss: 0 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 0 kB
Anonymous: 0 kB
LazyFree: 0 kB
AnonHugePages: 0 kB
ShmemPmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked: 0 kB
VmFlags: rd wr sh mr mw me ms pf io de dd
Here's the section for the memory mapped from the FPGA:
5555567000-5555588000 rw-p 00000000 00:00 0 [heap]
Size: 132 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 4 kB
Pss: 4 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 4 kB
Referenced: 4 kB
Anonymous: 4 kB
LazyFree: 0 kB
AnonHugePages: 0 kB
ShmemPmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked: 0 kB
VmFlags: rd wr mr mw me ac
I'm not sure if any of the differences in flags explains what is going on but I'm including them in case someone more proficient in memory mapping knows something.
