![]() (I can't say I've ever tried it with billions of files, I can say it works well into the thousands.) It does a really good job with large numbers of files and is way better than the built-in Windows copy utility. So if doing a block level replication isn't in the cards, you are going to have to live with the limitations of your file system's performance.Ī couple ideas, in increasing order of quality. Looks like you can do something like the following on NTFS: fsutil behavior set disablelastaccess 1 I don't know much about Windows, but one tweak we do on the Unix side is to disable logging file access time. There are also tweaks you can make to your FS to try to increase performance. Depending on how fast your storage is, you can parallelize your copy process by running more than one copy process on separate directories. This is a huge issue in life sciences, and it's something that's not really easy to find. This time adds up when you need to perform operations on millions and millions of files. ![]() I have hundreds of millions of files on an Isilon filer, and even with a pile of the highest performance nodes, metadata operations do still take at least a small amount of time. ![]() ![]() Your limitation here is filesystem overhead. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |