0

As far as I know the Linux stack size limit is 8 MB, but on 64-bit machines there's no reason why it couldn't be massively increased, e.g. to 4 GB. This would allow programmers to mostly not worry about storing large values on the stack, or using recursive algorithms instead of iterative.

It should also be faster because stack allocation is much much faster than heap allocation. We could see a whole new class of stack allocated data structures - imagine std::stack_vector<T> which allocates on the stack.

Is there a downside I'm not seeing? Or is it just that nobody has cared enough to make the change?

Timmmm
  • 527
  • 5
  • 15
  • I am just running a browser and a terminal, and I have close on 200 processes. You want to reserve 800 GB before I run any applications? Also, data on stack is transient -- it lasts only while the function that declared the data object is in scope. It can be done (ALGOL had no heap) but it was very constraining. On the other hand, my first mainframe had no stack. – Paul_Pedant Oct 12 '21 at 08:56
  • "You want to reserve 800 GB before I run any applications?" Yes - 800GB of virtual memory. There's no issue with that on 64-bit machines. – Timmmm Oct 12 '21 at 09:43
  • @ilkkachu: `std::vector<>` *always* stores its data on the heap. Only a pointer to the data and the size/capacity counters are stored on the stack. – Timmmm Oct 12 '21 at 09:44
  • 2
    Some consider 8MiB [to be huge](https://unix.stackexchange.com/q/473416/86440) ;-) – Stephen Kitt Oct 12 '21 at 10:25
  • @Timmmm, äh, right, sorry. I should have actually realized that if I'd thought it through (but I'm too much of a C coder for this). Anyway, even if you had "enough" space in the stack, it'd be hard to dynamically resize allocations there, so implementing `vector<>` on the stack probably wouldn't be that straightforward. But plain old static arrays could be used, and they already work. – ilkkachu Oct 12 '21 at 10:26
  • As Paul_Pedant said, stack allocations are transcient. Let’s have stack_malloc() function which allocates 1024bytes. The allocation is very simple : SUB ESP, 1024, but the return instructions start to undo the allocation (MOV ESP, EBP) in order to have a stack ready to return to the calling function. A `inline` function could however fix this issue (but this keyword just suggests not imposes an inlined function). And we would have to take into account that the stack allocated memory is freed as soon as the calling function returns which limits a lot the usefulness. – Frédéric Loyer Oct 12 '21 at 11:03
  • @ilkkachu Yes it would be harder to reallocate - you basically wouldn't be able to free the old allocation. But it could still be very useful! – Timmmm Oct 12 '21 at 11:19
  • @StephenKitt: It is huge *relative to existing limits that were decided for 32-bit systems*! – Timmmm Oct 12 '21 at 11:20
  • @Timmmm and tiny *relative to 32-bit systems with no stack limit*! (Such as AT&T Unix, and early versions of Linux.) – Stephen Kitt Oct 12 '21 at 11:27
  • Even if they did it wouldn't matter because it's *virtual memory* it doesn't actually use physical memory until you put something in it. – Timmmm Oct 12 '21 at 12:53
  • @Timmmm Page tables can be a scarce resource too. – Paul_Pedant Oct 12 '21 at 17:03
  • 1
    "using recursive algorithms instead of iterative" it only makes sense to use "recursive algorithms" with a language with mandatory tail-call elimination, like scheme. "Yes - 800GB of virtual memory. There's no issue with that on 64-bit machines" by that you're forcing the kernel to horribly *overcommit*, and when actual memory gets tight, the kernel won't be able to determine which commitments it should fullfill and which it should *drop*; the program which really needs to work with a large dataset at once, or the program whose author couldn't be bothered to learn how to program ;-) –  Oct 12 '21 at 18:09
  • But the clone(2) system call and library wrapper has a stack argument. Just use it -- and make the stack of the child process as large as you want. It won't be subject to `RLIMIT_STACK`. I don't think you will be able to mmap a 800GB chunk of virtual memory, though. –  Oct 12 '21 at 18:29
  • "it only makes sense to use recursive algorithms with a language with mandatory tail-call elimination" - yes but that's only because of the currently small stack sizes!!! – Timmmm Oct 13 '21 at 12:50
  • "you're forcing the kernel to horribly overcommit, and when actual memory gets tight, the kernel won't be able to determine which commitments it should fullfill and which it should drop", Linux already allows overcommit and kills randomly when out of memory. That could stay the same. – Timmmm Oct 13 '21 at 12:52

1 Answers1

1

While limiting stack size might originaly have been done to save virtual memory, another benefit is that it catches infinite recursions. Algorithms that require really deep recursion are not common, so if you're using an enormous amount of stack it's more likely to be a bug. This doesn't become less likely when switching to a 64-bit architecture.

An algorithm is likely to use more stack space in 64-bit mode because some data types are larger, but probably not so much that the stack size limit becomes a problem.

It's true that this limit also prevents allocating large arrays on the stack, which could be more efficient. But is fixing this important enough to lose the above infinite recursion check?

Barmar
  • 9,648
  • 1
  • 19
  • 28
  • Hmmm detecting accidental recursion seems like a reasonable advantage at least, thanks! – Timmmm Oct 13 '21 at 12:53
  • Edge-case I ran into: When working with Intel Fortran, automatic arrays (i.e. arrays declared with the size parameter being a runtime variable) and temporary arrays (e.g. the result of the expression `matmul(A, B)`) are stored on the stack, which a colleague benchmarked to have relevant impact on overall performance. Sadly, the implementation does this for arrays of *any* size, such that we ran into situations where `matmul(A,B)` was crashing the software under the default 16 MB stack size limit. Point being: Small default, combined with corporate ITs making it hard to change, can be an issue. – kdb May 18 '23 at 06:55