So, on Linux, either O_DIRECT
or O_SYNC
guarantees synchronous write, but which one has better performance?
This statement is not correct because as mentioned by @roland-smith on at least Linux, O_DIRECT
does not guarantee that the data has reached non-volatile media. It might happen to give that guarantee in a specific environment (e.g. writing directly to the block device representing a disk with a battery backed SCSI controller) but you can't rely on this in the general case (e.g. writing to a file in an ext4 filesystem backed only by a single SATA hard disk) because of at least the following:
O_DIRECT
on a file in a filesystem doesn't guarantee that the metadata necessary to retrieve the data back after a power loss crash has been written
- The kernel sent the I/O to hardware before the original call finished but the I/O is only in a volatile hardware cache
In the above scenarios sudden power loss would mean your program had thought I/O was successful when it had not. These days the Linux open(2) man page says this:
The O_DIRECT
flag on its own makes an effort to transfer data synchronously, but does not give the guarantees of the O_SYNC
flag that data and necessary metadata are transferred. To guarantee synchronous I/O, O_SYNC
must be used in addition to O_DIRECT
. See NOTES below for further discussion.
In the scenario given, on Linux the only way you to guarantee every write in synchronous write is to use O_SYNC
(which incur a speed hit) or do an fsync()
after every I/O (which is likely slower because you did two syscalls). If I was worried about speed I would forgo using O_SYNC
and would instead write in batches as big as possible and then fsync()
after a batch. Also be aware that if you're worried about data integrity you have check the return code of all fsync()
and all write()
calls (and close()
etc) for errors.
See this answer on "What does O_DIRECT really mean?" for further details and links.
On FreeBSD, to guarantee synchronous write, which option has the best performance: (1) O_DIRECT
+ fsync()
(2) O_DIRECT | O_SYNC
or (3) O_SYNC
alone?
You're in a similar situation to Linux (so see above) but of the three choices I would think the third one (O_SYNC
alone) would be fastest. The FreeBSD open(2) man page says this about O_DIRECT
:
O_DIRECT may be used to minimize or eliminate the cache effects of reading and writing. The system will attempt to avoid caching the data you read or write. If it cannot avoid caching the data, it will minimize the impact the data has on the cache. Use of this flag can drastically reduce performance if not used with care.
General note: using O_DIRECT
doesn't automatically mean all I/O will go faster - this depends on the workload (I/O size, I/O frequency, whether I/O is sequential or random, how frequently syncing is happening because it can impact merging etc) and how the I/O is submitted (synchronously vs asynchronously).