Friday, July 13, 2012

Large file upload more than 1GB using HTML5 Web workers, Drag and Drop and XHR2



To know about HTML5

This tutorial provides a guide and code examples for leveraging the large file upload using File API inside of a Web Worker. While we are trying to upload large file, we getting the problems like browser showing kill pages or unresponsive page etc.. This are because of we are blocking the user agent and we are utilising lot of CPU, it will lead to performance down or user agent getting continuos processing. We need to avoid the blocking or performance utilization on document. Solution for that is simple, we need to upload file using background processing nothing but using webworkers. By using webworkers we can upload files in background so that no errors or problems will occur.



In some cases reading the entire file into memory isn't the best option. For example, say you wanted to write an async file uploader. One possible way to speed up the upload would be to read and send the file in separate byte range chunks. The server component would then be responsible for reconstructing the file content in the correct order. Lucky for us, the File interface supports a slice method to support this use case. The method takes a starting byte as its first argument, ending byte as its second, and an option content type string as a third.

I was demonstrated the upload using file input and drag and drop. you could use client-side logic to verify an upload's mimetype matches its file extension or restrict the size of an upload. The most straightforward way to load a file is to use a standard <input type="file"> element. JavaScript returns the list of selected File objects as a FileList. Here's an example that uses the 'multiple' attribute to allow selecting several files at once. Another technique for loading files is native drag and drop from the desktop to the browser. This example include drag and drop support. The following example demonstrates reading chunks of a file.

If you load models or textures from external files, due to browsers' "same origin policy" security restrictions, loading from a file system will fail with a security exception. To solve this Start Chrome executable with a command line flag: chrome --allow-file-access-from-files or else run from local server


This will solve the following problems:
  1. How to upload large file more than 1GB
  2. Synchronous File Uploading
  3. Large File upload with XHR2 uisng webworkers
  4. Drag and Drop File upload
  5. Multiple File Upload without blocking UI
  6. Increasing performance While File uploads
  7. Background Processing of file uploads
  8. Slicing the file
code:
HTML

  <input type="file" id="files" name="files[]" multiple />
  <div id="drop_zone">
   Drop files here
  </div> 

  <output id="list"></output>
  


JavaScript


 var worker = new Worker('fileupload.js');
 worker.onmessage = function(e) {
 alert(e.data);
}
worker.onerror =werror;
function werror(e) {
  console.log('ERROR: Line ', e.lineno, ' in ', e.filename, ': ', e.message);
 }
function handleFileSelect(evt) {
 evt.stopPropagation();
 evt.preventDefault();

 var files = evt.dataTransfer.files||evt.target.files;
 // FileList object.
 
 worker.postMessage({
 'files' : files
 });
 //Sending File list to worker
 // files is a FileList of File objects. List some properties.
 var output = [];
 for (var i = 0, f; f = files[i]; i++) {
  output.push('<li><strong>', escape(f.name), '</strong> (', f.type || 'n/a', ') - ', f.size, ' bytes, last modified: ', f.lastModifiedDate ? f.lastModifiedDate.toLocaleDateString() : 'n/a', '</li>');
 }
 document.getElementById('list').innerHTML = '<ul>' + output.join('') + '</ul>';
}

function handleDragOver(evt) {
 evt.stopPropagation();
 evt.preventDefault();
 evt.dataTransfer.dropEffect = 'copy';
 // Explicitly show this is a copy.
}

// Setup the dnd listeners.
var dropZone = document.getElementById('drop_zone');
dropZone.addEventListener('dragover', handleDragOver, false);
dropZone.addEventListener('drop', handleFileSelect, false);
 document.getElementById('files').addEventListener('change', handleFileSelect, false);


Worker (fileupload.js) :

var file = [], p = true;
function upload(blobOrFile) {
 var xhr = new XMLHttpRequest();
 xhr.open('POST', '/server', false);
 xhr.onload = function(e) {
 };
 xhr.send(blobOrFile);
}

function process() {
 for (var j = 0; j <file.length; j++) {
  var blob = file[j];

  const BYTES_PER_CHUNK = 1024 * 1024;
  // 1MB chunk sizes.
  const SIZE = blob.size;

  var start = 0;
  var end = BYTES_PER_CHUNK;

  while (start < SIZE) {

   if ('mozSlice' in blob) {
    var chunk = blob.mozSlice(start, end);
   } else {
    var chunk = blob.webkitSlice(start, end);
   }

   upload(chunk);

   start = end;
   end = start + BYTES_PER_CHUNK;
  }
  p = ( j = file.length - 1) ? true : false;
  self.postMessage(blob.name + " Uploaded Succesfully");
 }
}


self.onmessage = function(e) {

for (var j = 0; j < e.data.length; j++)
  files.push(e.data[j]);

 if (p) {
  process()
 }

}


Conclusion
I hope this article has help you to upload large files. Web Workers are an underutilized and under-appreciated feature of HTML5. We can minimize the work to upload a large file. The technique is to slice the upload into multiple chunks, spawn an XHR for each portion, and put the file together on the server. This is similar to how GMail uploads large attachments so quickly.

Further Reading:
  1. Web Socket - Web RTC video conference.
  2.  Mastering in webworkers 
  3. Reading files and chunking. 
  4. HTML5  

19 comments:

  1. Hi,
    Thank you for this example.
    Could you share the server script as well? I don't really get how to save the file server-side.
    Thank you in advance,

    Valerie

    ReplyDelete
    Replies
    1. I will share php script in a quick session

      Delete
  2. While this is a great overview, I wasn't able to implement due to a few bugs.

    ReplyDelete
  3. Bro...change globally theme..it's hard read using mobile browsers

    ReplyDelete
  4. Bro...change globally theme..it's hard read using mobile browsers

    ReplyDelete
  5. consloe.log - in first javascript - must be console.log,
    expect that little thing - Great Post
    thanks

    ReplyDelete
  6. Updates: webkitslice prefix removed so you can use as it as in specification.

    ReplyDelete
  7. Hi. Thanks for sharing.

    Dunno if I miss something but when I select file, I get this error in console "DataCloneError: The object could not be cloned.
    'files' : files"

    I search on the web but don't really find what can be the reason of this error.

    As previously ask, could you share a server-side script (PHP ?) to rebuilt file on the server.

    Thank you in advance.

    ReplyDelete
  8. Great article.

    Can you provide the php script to try this on my computer and test how much memory will consume?

    Thanks in advance

    ReplyDelete
  9. Great Article,

    Hi can you please send the php script.

    ReplyDelete
  10. Great! Could you please share the php script? Thanks in advance, Jan

    ReplyDelete
  11. It would be nice if you could attach the working project for reference.

    ReplyDelete
    Replies
    1. okay, I would Place working project ASAP.

      Delete
  12. Can you please share how to save the file on the server side?

    ReplyDelete
  13. please share how to save the file on server side?

    ReplyDelete
  14. please share how to save this file on server particular location i.e. shared location on server?

    ReplyDelete