I've tried running QRecall on Amazon SSS, and failed. Do you know why it failed (it may be something to do with the dynamic file allocation) or is there another online service that would work? (dotMac didn't work either). I want to use QRecall for my home user clients. I fear that I will have to have a server here for them to use, but I'd rather not.

I've also tried dedicated online storage firms, and had a succession of failures. maybe it's me (don't answer that)

I've never used Amazon SSS, so I can't comment about that. I have used QRecall on .Mac and it works ... at least for small amounts of data. Any remote storage solution like .Mac is going to be seriously hampered by the data transfer rate, which is orders of magnitude slower than even networked volume on a local area network, which itself is going to be 10 times slower than a external hard drive. So, I can imagine it working for a few dozen megabytes of data. If you just want to capture a modest number of user documents, it should work.

I'd be curious to see what problems you're having, especially any error messages in the log file. One, all too common, problem with remote storage systems is their inability to deal with files greater than 4GB.

There are also a number of Network Area Storage (NAS) devices, which are essentially self-contained file servers or drive sharing devices, that you connect to your network.

A less expensive solution is an external Firewire drive. You can configure QRecall to automatically backup to the external drive whenever the drive is plugged in. Then, just periodically move the drive from one system to another for regular backups.

You can always use personal file sharing to set up one system as a file server and have the other computer backup to that.

I'm currently backing up to a NAS, and so far it's taken 13 hours to do 7 Gb out of 130. The problem with Amazon SSS is that QRecall kept stopping working. sorry that's not more specific, but it did seem to get confused.

How close are we to a release version and do you have any ideas on costing yet? My initial trials were most impressive, though I find the interface confusing: I'm not sure I'd want to let all my users loose on it, given the problems they have understanding Retrospect.

Andrew Daws wrote:I'm currently backing up to a NAS, and so far it's taken 13 hours to do 7 Gb out of 130.

I'd be curious to learn what the network configuration is. Is this a remote storage via a DSL or Cable connection, or this on a LAN? Ethernet or WiFi?

The problem with Amazon SSS is that QRecall kept stopping working. sorry that's not more specific, but it did seem to get confused.

Zip up the log files in ~/Library/Logs/QRecall and either e-mail them to me or post them in the forum. There could be some clues as to what's going on.

How close are we to a release version and do you have any ideas on costing yet?

I initially planned to have the final release ready last month, but a series of Leopard compatibility issues has pushed that out. Baring any more castrophic bugs, I plan to have the first commercial release of QRecall ready by MacWorld in January.

An identity key is $40. You will need at least one key to capture data.

A household (all of the people/computers that share a primary residence) are permitted to reuse a single key to capture data to separate archives.

You will need multiple keys if you want to capture data from two or more computers to a single archive. This maximizes QRecall's ability to eliminate duplicate data in the archive. The unique identity keys on each system allow QRecall to organize the data in the archive by owner.

Multiple keys are required for business or institutional users, even if systems are captured to separate archives.

The NAS is on the local network, so I don't know why it's taking so long. Indeed backup up to Amazon SSS was very fast, which is why I got so excited initially.

I want to have a NAS at home available on the Net (I have a batch of fixed IP addresses) but I haven't managed that so far. Must be something to do with the router. Any helpful hints on that would be much appreciated.

$40? That's good news. I'm sure I could get several of my clients to stump up that much. Might there even be some kind of reseller discount???

I think I've attached the log files

The majority of the problems I see in your log are I/O errors. It appears that QRecall is simply losing connection with the server that it's writing to. As you've discovered, this has rather catastrophic consequences for the data integrity of the archive.

If Amazon SSS is anything like .Mac (I have no experience with Amazon SSS), it will appear to be blindingly fast at first. It's an illusion. Because of the slow transfer speeds of home Internet connections, these drivers buffer massive amounts of data locally. So you can write megabytes of data to the drive and it appears to be written instantly. But when it's time to close the file or disconnect the volume, it hangs for 20 minutes while it finishes sending the data.

I don't know why a local NAS volume would be that slow. Have you tried simply copying some very large files to the device? (QRecall archives make particularly good test subjects.)

$40? That's good news. I'm sure I could get several of my clients to stump up that much. Might there even be some kind of reseller discount???

Not at the moment.

James Bucanek wrote:I've never used Amazon SSS, so I can't comment about that.

You might want to take a look at it. The "Security Now!" just devoted an entire episode to Amazon S3 and JungleDisk, which is a Mac-compatible front-end to S3. I don't know if that's what the original poster was using or not, but it gives you access to S3 that looks like any other mounted network volume.

I agree that unless you've got really really fast Internet uploading speeds (and note that most cable and DSL services cap your upload speed to a small fraction of your download speed) you won't want to back up anything large to S3, at least not frequently. But I can see where it would have its uses, especially if you have a few critical things you want to store off-site, and it might be nice to use QRecall for that.

I should add that I haven't tried S3 myself yet, but it's on my list of things to do.

Yes I am using Jungle Disk, and I'm getting an upload speed in excess of 1.3 Mbps. The problem in both that and a locally attached NAS (I've tried 2) is that the archive file gets corrupted, and will not repair. I've lost count of how many new archives I've set up: when I try to repair one it says it's doing it but never does. I suppose I could give up and use it only on USB drives, but I really want my clients to back up online so that I don't have to visit them to troubleshoot!

Andrew Daws wrote:when I try to repair one it says it's doing it but never does.

Post, or e-mail me, your log files and I'll try to determine what the problem is.

After a couple of days of experimentation and research, my opinion is that Amazon's Simple Storage Service (SSS) and QRecall are profoundly mismatched.

Amazon's SSS is based on WebDAV. WebDAV is a web server based file transfer protocol originally designed to update web sites remotely. Amazon SSS, and I suppose any similar WebDAV based system, has tried to use WebDAV to emulate a remote file system.

The fundamental problem is that WebDAV can only transfer files. When you open a remote file, some intermediate agent (JungleDisk, in this case) copies the entire file from Amazon's servers to your local hard drive. When you write and close the file, the entire file is uploaded back to the server. For small files this is great. For large file, the overhead becomes ridiculous, particularly for small changes.

And that's exactly how QRecall works: It makes small changes to very, very, large files.

As an experiment, I created an Amazon SSS account and installed JungleDisk. I created a new archive and captured my Movies folder. This just happened to have about 2.3GB of data in it - perfect for checking for 2GB file size problems on SSS.

The capture completed in less than a minute, since all of the data was initially written locally. When QRecall closed the new archive file, JungleDisk began uploading the repository.data file to the server. My outgoing data is capped at 600Kbps, so 2.3GB takes about 9 hours to transmit.

After 8 hours I checked on it again. There was a "Background Operation Retry" message in the errors and warning pane. JungleDisk was in the process of re-uploading the entire 2.3GB data file a second time. This repeated at least two more times. 38 hours later, JungleDisk finally finished uploading the data to the server. This is not particularly surprising, as the chances of getting a data transmission error in a single file increase exponentially with the size of the file.

I made a small change to a single file and recaptured the Movies folder. When QRecall opened the archive, JungleDisk downloaded the entire 2.3GB archive. This took about a half hour. The capture then ran (about 15 seconds), and QRecall closed the archive. JungleDisk then began uploading the entire 2.3GB archive back to Amazon.

So adding 1.5MB of new data to the archive (a change of less than 0.06%) resulted in 4.6GB of data transfer and took over 9 hours. Not only is this outrageously inefficient, but it's expensive too. Amazon charges between $0.10 and $0.18 per GB transferred. If you updated the archive every day, it would cost approximately 50 times the $0.15/GB monthly storage fee. For this example, the 2.3GB archive would cost only $0.35 to store, but would be hit with $17.50 of data transfer fees.

Some searching on the JungleDisk support forums confirmed these observations. JungleDisk only transfer whole files. It cannot update only a portion of a file, and it cannot retransmit just the portion of a file that has failed to transfer (although the latter is feature they want to add). This makes large file transfers both brittle and time consuming. Amazon and/or JungleDisk currently have a hard 5GB file size limit. In QRecall terms, 5GB is a small archive and certainly smaller than the backup storage requirements of most users.

My conclusion is that there is a serious impedance mismatch between QRecall and Amazon SSS - and probably any WebDAV based remote file system emulator. I suppose this includes .Mac, but I'll have to experiment.

QRecall is terribly efficient at minimizing the storage required to keep multiple incremental captures. Amazon SSS could complement QRecall by using it to mirror a local archive (i.e. uploading the archive once a week or month as an off-site backup). But even in that scenario, the 5GB file size limit would quickly become an issue.

Let me follow up that rather pessimistic post with something a little more upbeat.

I have been considering a number of changes to the way QRecall stores and organizes files that could mitigate some of the issues with SSS. Some of these changes are being considered for new redundancy and data journaling features currently under development.

But I've also been toying with some king of "archive archive" feature that would split an archive into pieces suitable for writing to a series of DVDs, or maybe an on-line storage system like Amazon's SSS.

It wouldn't be the same as using the archive remotely. But having a built-in method of exporting an archive to off-line storage and then recovering it again might go a long ways towards making services like SSS more attractive for archive redundancy.

Fantastic. Well done for doing that research.

I suggest that I bin the current log files, and concentrate on my problems with a NAS. And if a NAS does work locally, would I be able to make that available to my clients?

And would it still suffer from this problem of uploading and downloading the entire file?

Andrew Daws wrote:I suggest that I bin the current log files, and concentrate on my problems with a NAS. And if a NAS does work locally, would I be able to make that available to my clients?

And would it still suffer from this problem of uploading and downloading the entire file?

NAS fall into two types — remote access hard disk (single user) or a hard disk and file server in a box (multi-user) — neither of which will have any of the issues associated with a WebDAV based service.

I've given up. I reinstalled my system, and set up a QRecall archive on the same hard drive. The first time it backed up fine, then after that it stuck on opening archive, and when I clicked cancel (which at least it did recognise this time) it said it had found nothing to back up.

Maybe it's a Leopard issue, maybe it's not yet ready for public consumption, but I'm very disappointed that this software doesn't live up to its considerable promise.

Andrew Daws wrote: The first time it backed up fine, then after that it stuck on opening archive, and when I clicked cancel (which at least it did recognise this time) it said it had found nothing to back up.

That's exactly what it would say if you canceled the capture before it had captured anything.

Maybe it's a Leopard issue, maybe it's not yet ready for public consumption, but I'm very disappointed that this software doesn't live up to its considerable promise.

As I've said, I'd love to see log files. I can only correct the problems I can diagnose.