This article features our recent experience with multipart file upload to AWS S3. Our application builds line-delimited JSON files that contain patient information and uploads them to an AWS S3 bucket. The file consists of a few hundred lines, each representing a patient record.
Since each row was up to 10MB, we adopted the "multipart S3 file upload" approach. This method allows us to break the file into parts and upload them using ThreadExecutive, ensuring good performance. In this approach, we are required to pass the part stream and its byte size as arguments in the Part Upload Request.
The multipart AWS S3 upload was successfully done, as anticipated. However, after going live, we encountered a production bug. The uploaded file had an incomplete line in the middle, with a few characters missing at the end. The issue occurrence was sporadic and unreplicable.
Upon investigation, we discovered that the issue occurred when the patient record had a binary file-like image attached. Further inspection revealed something unusual with the highlighted line.
After replicating the issue in our Dev environment with a suitable record, we changed the following lines and resolved the issue.
We changed content.getBytes().length to inputStream.available().
We found that content.getBytes().length minus inputStream.available() yielded 0 with normal string contents and -1 or -2 with binary strings, such as images. This discrepancy occurred with a few images but not all.
The documentation notes, "Note that while some implementations of InputStream will return the total number of bytes in the stream, many will not." Therefore, InputStream.available() might not be appropriate in all cases, which is why we initially decided against using it.
Conclusion
Switching from content.getBytes().length to inputStream.available() addressed the issue of incomplete lines in our S3 multipart uploads, especially when dealing with binary files like images. This change was crucial in stabilizing the upload process. The problem arose due to the method's inconsistent handling of binary data, as noted in the documentation. This experience underscores the importance of thorough testing and selecting the right methods based on the nature of the data to ensure reliable file uploads.
.png)
.avif)
.avif)
.avif)
%20Top%20AI%20Agent%20Frameworks%20for%20Autonomous%20Workflows.avif)
%20Understanding%20the%20Role%20of%20Agentic%20AI%20in%20Healthcare.png)
%20AI%20in%20Software%20Development_%20Engineering%20Intelligence%20into%20the%20SDLC.png)
%20AI%27s%20Role%20in%20Enhancing%20Quality%20Assurance%20in%20Software%20Testing.png)
.png)
%20Tableau%20vs.%20Power%20BI_%20Which%20BI%20Tool%20is%20Best%20for%20You_.avif)
%20AI%20in%20Data%20Quality_%20Cleansing%2C%20Anomaly%20Detection%20%26%20Lineage.avif)
%20Key%20Metrics%20%26%20ROI%20Tips%20To%20Measure%20Success%20in%20Modernization%20Efforts.avif)
%20Hybrid%20Cloud%20Strategies%20for%20Modernizing%20Legacy%20Applications.avif)
%20Harnessing%20Kubernetes%20for%20App%20Modernization%20and%20Business%20Impact.avif)
%20Monolith%20to%20Microservices_%20A%20CTO_s%20Decision-Making%20Guide.avif)
%20Application%20Containerization_%20How%20To%20Streamline%20Your%20Development.avif)
%20ChatDB_%20Transforming%20How%20You%20Interact%20with%20Enterprise%20Data.avif)
%20Catalyzing%20Next-Gen%20Drug%20Discovery%20with%20Artificial%20Intelligence.avif)
%20AI%20Agents_%20Digital%20Workforce%2C%20Reimagined.avif)
%20How%20Generative%20AI%20Is%20Revolutionizing%20Customer%20Experience.avif)
%20Leading%20LLM%20Models%20Comparison_%20What%E2%80%99s%20the%20Best%20Choice%20for%20You_.avif)
%20Generative%20AI%20Strategy_%20A%20Blueprint%20for%20Business%20Success.avif)
%20Mastering%20LLM%20Optimization_%20Key%20Strategies%20for%20Enhanced%20Performance%20and%20Efficiency.avif)
.avif)
.avif)
.avif)
.avif)
