How can I analyze a large amount of data sent to my server before writing it to disk?

Question

Note: this is a follow-up to this question.

I am developing a website that allows users to upload large chunks of data to my Java backend. Once validated, the data must be stored on disk. Here, "large" means in the order of 10 KB up to 100 KB.

Given that a large number of users can submit large chunks of data at the same time, and that the data must first be analyzed (validated) before being stored on disk on the server side, what options do I have to prevent JVM out-of-memory errors while processing the data chunks? Note that each data chunk is received at the moment from the frontend in a String object.

Answer 1

Write to a temporary file first

Having a server crash because you are running out of ram is an avoidable problem. Writing to a temp file allows you to get all the chunks in and analyze at once, or analyze each chunk when it comes in.

Make sure the file is deleted when you are done with your evaluations. If it passes muster, you can move the file to its permanent location with its correct filename. A filesystem move is usually just rewriting a pointer on disk so it's incredibly fast.

Answer 2

I would echo @Berin Loritsch concerns about using memory for the same reasons mentioned.

However, I would store the data in a database and do my analysis from there.

I am probably going to start a firestorm debate (unintended) but generally speaking, databases handle sorting, searching, auditing, logging, error handling, data manipulation, security, storing, etc. better (faster, easier, more secure) than the file system, at least in most cases. Databases offer an easy means to relate and retrieve stored data using SQL.

You can use temporary tables for your operations and commit the (valid) data to the database after your analysis phase.