<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi Clemens,</div><div><br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
I see you're getting the hang of the difficulties of the project now :)<br></blockquote><div><br></div><div>That’s really encouraging for me to know that you think so ^_^ </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
You have the right ideas to solve the problem. Do keep in mind though<br>
that CAS will only work up to a word size or a double word size at most,<br>
i.e. swapping more than 64 bit with CAS atomically is probably not going<br>
to work.<br></blockquote><div><br></div><div>Got it. I may swap one by one instead if swapping the complete structure and its just 4 variables.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<br>
I'm not sure whether we will actually need a separate process to<br>
increase the allocation size. Consider this proposal:<br>
<br>
struct shared_block_mgmt {<br>
size_t last_known_size;<br>
size_t refcnt;<br>
int blockfd;<br>
struct shared_block* block;<br>
};<br>
<br>
struct shared_block_mgmt block_mgmt;<br>
<br>
struct shared_block {<br>
size_t size;<br>
size_t used;<br>
char memory[];<br>
};<br>
<br>
This struct would mean that the first `used` bytes of `memory` are<br>
in-use by our data structure and everything after that up to `size`<br>
bytes is free[1].<br>
<br>
Any allocation request where used + request_size <= size would succeed<br>
and we could change `used` using CAS to ensure that this allocation<br>
operation is atomic.<br>
<br>
Any allocation request where used + request_size > size would trigger<br>
growing the memory area. Growing would calculate the size we want to<br>
grow to, let's call this `target_size`. Then, it would attempt to grow<br>
atomically using:<br>
<br>
size_t local_size;<br>
size_t local_used;<br>
size_t target_used;<br>
size_t target_size;<br>
bool needs_resize;<br>
do {<br>
local_used = block_mgmt.block->used;<br>
local_size = block_mgmt.block->size;<br>
target_used = local_used + request_size;<br>
needs_resize = target_used < local_size;<br>
if (needs_resize) {<br>
// growing required<br>
target_size = local_size + BLOCK_GROWTH;<br>
ftruncate(block_mgmt.blockfd, target_size);<br></blockquote><div><br></div><div>What I was doing is to keep a file created beforehand and append it whenever needed to the existing file.</div><div>ftruncate() is much better & cleaner approach I guess.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
}<br>
} while (<br>
(needs_resize &&<br>
!CAS(&block_mgmt.block->size, local_size, target_size) &&<br>
!CAS(&block_mgmt.block->used, local_used, target_used)) ||<br>
(!needs_resize &&<br>
!CAS(&block_mgmt.block->used, local_used, target_used)));<br>
<br>
// At this point, either resizing was not needed and block->used is<br>
// what we expect it to be (i.e. allocation succeeded), or resizing<br>
// was needed, and we did successfully resize, and after resizing we<br>
// did update block->used (i.e. allocation also succeeded).<br>
<br>
This will oportunistically call ftruncate with the bigger size on the<br>
mmap'd file descriptor, but calling ftruncate in multiple processes at<br>
the same time should not be a problem unless the truncation would ever<br>
shrink the file (which we can not completely avoid, but make very<br>
unlikely by making BLOCK_GROWTH</blockquote><div> </div><div>Maybe, we can call ftruncate outside the loop and just set size first.</div><div>I dunno if it is possible but if it is we can modify compare and swap to behave in the following way :</div><div>If the process detects that used + request_size > size and area needs to be grown, instead of swapping the size with </div><div>its new value, we can just increment the old value without caring what is the old value by ‘requested_size’.</div><div>Now in this type of swapping, we just don’t need to care what is old value, all we do is increment requested size to it.</div><div>So if 2 or more process do this simultaneously, they just end up incrementing the memory correctly. And to keep a margin we can add some extra_growth to each.</div><div>Although I m not sure if atomic operations allow this,</div><div>a better approach maybe to just share a variable called maxFileMemory which can only be swapped if the new value being provided is greater.</div><div>Before mmaping for write or read we may just ensure if current file size matches maxFileMemory.</div><div>This maybe would prevent shrinking.</div><div><br></div><div>Or maybe we somehow can block shrinking by implementing our own ftruncate or some other way maybe.</div><div><br></div><div>I guess I will need to think of them more briefly about cases and everything to get it in right words I guess.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"> sufficiently large so that any pending<br>
extension requests would be processed before another extension is<br>
required). </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"> <br></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
You correctly identified that we'd have to re-mmap(2) the file after<br>
this operation, which is why I've included last_known_size in struct<br>
shared_block_mgmt. If last_known_size differs from the size stored in<br>
struct shared_memory, that would trigger creation of a new struct<br>
shared_block_mgmt with a fresh mmap with the larger size. We cannot<br>
unmap the old mapping until all threads are no longer using it, so we'd<br>
have to keep a reference count (field refcnt). And finally, we'd<br>
probably have to check whether any offsets in the data structure are<br>
larger than last_known_size, and if so, retry the current lookup<br>
operation with a fresh mmap(2) of the shared memory area, since it must<br>
have grown during the pending lookup.<br></blockquote><div><br></div><div>Yes true and actually this way implementing it makes it more precise and clean.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<br>
I believe this would allow us to avoid the separate process to grow the<br>
shared memory and also not require the use of a spinlock to block all<br>
processes for the resize operation. What do you think?<br></blockquote><div><br></div><div>That’s right, this will do it in a single process.</div><div><br></div><div>The reasons for me creating a new process were:</div><div><br></div><div>a) The process expanding the memory can die in between because we dunno about that process</div><div>Although the above approach would make the other process enter a loop of creating memory automatically, so this removes the need of separate process.</div><div><br></div><div>b) As I was creating a brand new file which was to be appended, if many build process tried to create more memory simultaneously, the last process which creates that new file would actually create that and if the any other processes of the many existing ones were gonna append the file of what they think it was, would end up appending the one created by last process and not actually appending the file they desired. </div><div><br></div><div>c) It mainly was intended to give more control over the tasks.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<br>
[1] This memory management approach may actually not work for us later<br>
on and we may have to use a free-list or buddy system with a bitmap, but<br>
for the point of discussing the extension, let's simplify the memory<br>
allocation for now.<br></blockquote><div><br></div><div>Got it! </div><div><br></div><div>Tomorrow I will try to analyse all the cases on this implementation so if any downsides are left, they can be removed and I will update you with a pdf.</div><div><br></div><div>Regards,</div><div>Mihir <br></div><div><br></div><div><br></div><div> </div></div></div></div></div></div></div>