In general, I guess S3 will store the metadata in something HBase, where the object names are sorted, and a list of a directory is just a prefix match, so if we just listing the tail elements of a path, it will be much faster than listing the head elements, as we will only scan all the things we want, while listing the head elements, we need to scan all the objects...

I guess I should more clearly state my concern with this approach. The main reason to remove rename from the critical processes is to reduce the burden on the filesystem/store and improve performance. If as a part of writing, we need to read and write another metadata file, I don't see how we are going to improve performance by that much or reduce the burden on the filesystem.

Do you have any perf numbers from a prototype that show this is beneficial?
Why choose a file rather than using something we already guarantee is fast such as a being backed by a WAL?

I think regardless, this is an optional implementation, not the default, so I think it's okay to implement, but I would have these concerns if we ever were considering changing the default.

This is for the store file. We still need to support writing store file at another place(tmp directory, or just another place when bulkloading), and then commit it by renaming it to the store file directory.

Yes, if it can be parsed to a pb object then it is fine. If not, we just skip to the next one.

Sorry, what do you mean by toggling?

I meant does HBase control setting this config (it knows it should fall back to loading again) or the user needs to configure this.

Remove the broken store files. As now we write to store file directory directly, a region server crash in the middle of a flush and compaction could lead to partial store files in the store file directory, the old code can not deal with this situation.

We want the storage system to be self maintained. As we rely this storage system to store our region data, if it depends on a region, it will introduce cyclic dependency. I will not say this is impossible, but I just do not like this style of architecture. Performance comparing about write a new metadata file and write a hbase row is not the most important thing here, as they are all much faster than reanming the actual data file on S3 right? This is the problem we need to solve here.

What if we just crash before writing the empty file but after we move the file to store directory? Still the same problem?

The bulk load is not successful until the write of the empty file

But we will still remove the file when opening the region again?
Or maybe we need to redesign the bulk load process to make it transactional? Like a procedure?