I think you should put your processor above Sitecore's processor, as they run in order. In your case it should be:
<formUploadFile>
<processor type="scwffm2.Helper.LimiteFileSize, scwffm2.Helper"/>
<processor type="Sitecore.Form.Core.Pipelines.FormUploadFile.ResolveFolder, Sitecore.Forms.Core"/>
<processor type="Sitecore.Form.Core.Pipelines.FormUploadFile.Save, Sitecore.Forms.Core"/>
</formUploadFile>
Another option would be to check the filesize using jQuery, making sure you can't even upload the file in the first place:
$.validator.addMethod('filesize', function(value, element, param) {
// param = size (en bytes)
// element = element to validate (<input>)
// value = value of the element (file name)
return this.optional(element) || (element.files[0].size <= param)
});
$('#inputid').validate({
rules: { input: { required: true, filesize: 1048576 }},
messages: { input: "File must be less than 1MB" }
});
Please note that I haven't tested this code, so it might need some tweaking