Upon reviewing my typical workflow, I realized that perhaps I could more efficiently structure my drive arrays and here is what I came up with. I'm not settled that it is an ideal solution, but for right now is working better than my previous solution.
My consideration: Striping additional drive arrays for performance, and the best way to implement it?
Problem: I/O conflicts from reading and writing large files to the same drive array as when converting raw files and saving them back to a shoot folder. So ideally, we should probably read raw files from one disk and write processed files to a separate disc, all while our OS is residing on yet its own disk -- in this, there are no clogs while a single drive has to compete for I/O time or compete with any OS paging, though OS paging is probably a minor concern on fast systems with large amounts of RAM. From prior discussions, I know several of you already do this at least on single drives and have commented on notable time savings while doing large volume batch processing. I was reading and writing from and to a single 2-drive striped 'working image' array and getting results comparable to when I read from a single drive and wrote to a separate single drive, so I left it alone until now...
My current implementation uses a pair of 2-drive RAID 0 arrays. The first array has all my working image files, including the raws from the shoot. The second contains my OS and a large amount of free space for the desktop. As I read the raws from the working image array and process to the desktop array, I find batch conversions to be significantly faster now that I process between these stripes; I now have sustained throughputs between these arrays at 150MB/s which is double what I had before when one or both sides were on a single drive, or when both were on a 2-drive stripe. (PS: I do copy the folder of converted files back into the folder for the shoot with the raws for permanent storage, and of course realize that adds total time, but I schedule this when the system is otherwise idle. And of course this is regularly backed up redundantly to the Drobo.)
What I'm not sure of, is would I have been better off, worse off, or about the same if I simply ran all 4 drives as a still faster stripe and lived with any I/O bottlenecks? I know, I could set it up and try it, but it seems a bother, especially if I end up the same or slower
Any thoughts?
My consideration: Striping additional drive arrays for performance, and the best way to implement it?
Problem: I/O conflicts from reading and writing large files to the same drive array as when converting raw files and saving them back to a shoot folder. So ideally, we should probably read raw files from one disk and write processed files to a separate disc, all while our OS is residing on yet its own disk -- in this, there are no clogs while a single drive has to compete for I/O time or compete with any OS paging, though OS paging is probably a minor concern on fast systems with large amounts of RAM. From prior discussions, I know several of you already do this at least on single drives and have commented on notable time savings while doing large volume batch processing. I was reading and writing from and to a single 2-drive striped 'working image' array and getting results comparable to when I read from a single drive and wrote to a separate single drive, so I left it alone until now...
My current implementation uses a pair of 2-drive RAID 0 arrays. The first array has all my working image files, including the raws from the shoot. The second contains my OS and a large amount of free space for the desktop. As I read the raws from the working image array and process to the desktop array, I find batch conversions to be significantly faster now that I process between these stripes; I now have sustained throughputs between these arrays at 150MB/s which is double what I had before when one or both sides were on a single drive, or when both were on a 2-drive stripe. (PS: I do copy the folder of converted files back into the folder for the shoot with the raws for permanent storage, and of course realize that adds total time, but I schedule this when the system is otherwise idle. And of course this is regularly backed up redundantly to the Drobo.)
What I'm not sure of, is would I have been better off, worse off, or about the same if I simply ran all 4 drives as a still faster stripe and lived with any I/O bottlenecks? I know, I could set it up and try it, but it seems a bother, especially if I end up the same or slower
Any thoughts?