[This article was first published on JottR on R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Fit a cross-validated elastic net with cv.glmnet(x, y) on a design matrix that comfortably fits in memory, and the call can still fail. The reason is that the function needs several times the size of x while it runs, and nothing anywhere in your script says so. This post is about giving that requirement a home – a memory specification, written in R next to the code that knows it – first for a plain sequential call, then for the same call running in parallel. This post describes experimental ideas and future plans for the Futureverse ecosystem. With one exception – ‘futurize()’, which is on CRAN today – the features shown here are not yet implemented.TL;DRWrite down what the call needs, as one annotation on otherwise ordinary code;fit resources(memory(4 * object.size(x)))Better still, the memory specification need not stay at the call site. Once cv.glmnet() carries its own declaration, parallelization frameworks can make use of it, e.g.fits futurize()Either way, that one declaration is meant to do two jobs;checked before the work starts – parallel or not – it fails fast instead of minutes or hours later, andhanded to a parallel framework, it also decides how many tasks can run in parallel.The goal of the Resource Project is to study, design, and implement these features in the Futureverse. Feedback and suggestions are welcome.Problem: We have no way of declaring memory needsWe check arguments all the time. A stopifnot(is.matrix(x)) at the top of a function is second nature and in our muscle memory. It is useful because it helps functions fail fast when the wrong arguments are passed, and because we can give an informative error message. However, we have nothing for checking whether the machine is actually capable of running the function.For example, you might not have enough memory available to perform a calculation. If you run out of memory, you might get:> fit Error: freeMemory() >= 4 * object.size(x) is not TRUEThat fails instantly and before attempting the model fit, if there is not enough memory available. We could also imagine a richer vocabulary that provides us with more informative error messages, e.g.assert_resources(memory(4 * object.size(x)))#> Error: UnmetResourceError: requires memory 7.5 GiB, available 3.1 GiBIn this project, we’re proposing an expr |> resources(...) syntax for declaring and asserting resource needs, including memory requirements, next to the code where it applies. In our example, it would look like:fit resources(memory(4 * object.size(x)))#> Error: UnmetResourceError: cv.glmnet(x, y) requires memory 7.5 GiB, available 3.1 GiBThis syntax preserves the original code and logic as-is, while allowing you to declare resource requirements that R can act on. In its most basic form, it effectively works like:fit futurize()the future framework could work together with future.batchtools to translate each of the calculated resource needs into declarations understood by the job scheduler, which then can find appropriately sized slots on the cluster - all while maximizing the memory use but without ever running out of memory.Ideally, everything is hidden awayJust as with resources(), if cv.glmnet() declares its own resource needs, futurize() can also take advantage of that. That would close the circle such that code existing already today, e.g.fits futurize()would become resource aware, protect against overuse, and optimize scheduling overnight - all without code changes.OutroPhew, that was quite long, and yet, I only got to cover a tiny bit of what the Resource Project aims for. I discussed how we can manage memory from within R, but there are many other compute resources that limit us. For example, we also want to manage walltime, scratch space, GPU cores, and GPU memory.If you have other thoughts or ideas, we’d love to hear from you. Please reach out on the Futureverse Discussions forum.May the future be with you!HenrikA better memory model for cv.glmnet() has the form a + b * object.size(x) + c * ncol(x): a fixed cost a that dominates while x is small, the b * object.size(x) term used above, and a c * ncol(x) term for per-column bookkeeping that only becomes visible on very wide matrices. Working out the coefficients, and how to measure them, is a project in itself. Once x is large, b * object.size(x) becomes the dominant term. [return]The resource function arguments should match that of the function. Because of that, we could generate those automatically and simplify the setter to just be a quoted expression, e.g. quote(resources(memory(4 * object.size(x)))). [return]It might be that static-code inspection can be used to avoid having to declare resources(cv.glmnet) and instead just use fits resources(). [return]Called without arguments, assert_resources() queries sys.function() for the function currently being evaluated, takes its "resources" attribute, and calls it with the arguments of the call in progress. [return]To leave a comment for the author, please follow the link and comment on their blog: JottR on R.R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.Continue reading: The Resource Project: Tell R How Much Memory You Need