Plotting and Mining on Amazon Cloud Drive
Have been plotting and Mining on ACD for a few Weeks now, some success and a lot of problems, most of which still need further work. I will share what I have found, others please do the same.
First off to answer a couple of questions from Another thread.
Thanks for your response. Great question. I just ran speedtest.net and got 10.3 Mbps upload speed. Currently on wifi. Will try again over LAN.
What size files are you uploading? I'm using the desktop ACD upload app. Is that what you're using?
Thanks for any help you can provide!
10.3 Mbps is only 1.03MB/s so that will be the limiting factor in your upload speed.
Recently I have also been uploading in 25GB chunks as that is the biggest size I find to give me around 90% success rate.
50GB is the maximum for ACD and I have done a number of uploads of 48GB but with around 50% success rate, sometimes a string of files go up other times I get a number of failures in a row, hence why I have after some experimentation dropped down to 25GB.
I suspect if I had a faster Upload Speed (currently 2MB/s at best) the larger files would be more reliable?
I have tried the Amazon Desktop Ap, but not with much success. Have also used NetDrive2 for the Upload which is better. Lately I have been using rclone which I have found the best so far.
Apart from Upload speed the biggest Upload problem is that with larger files there is an increasing delay between the file getting to ACD and when it shows in the directory, which causes all sorts of problems.
Next Credit and Links to other Topics on Cloud Mining that inspired me to give it a go.
Well worth a read.
So as I see it there is one Big problem and one Big potential problem with ACD.
The Big Problem is what bandwidth you have. I only have 2MB/s upload which means that it takes about a Week to upload 1TB. 20MB/s would mean 10TB a Week which I think would be OK.
Then there is the Download problem when reading the plots. I have 8MB/s download and usually this is maxed out when a new Block comes up. However with the 3.4TB of Plots I have the read time at that speed is already way too long, and often the next block is there before the read has finished.
On top of that not always do I seem to max out my Bandwidth either on Upload or Download, so I think that on occasions Amazon are restricting the bandwidth. I have run Speedtest on some of these occasions and always get the expected speeds. So I would say that idealy you need a Download Bandwidth of 100MB/s for any decent sized plot.
Then there is the potential problem. That is that shortly after you have got 50TB happily mining that Amazon takes a dislike to this mode of use and you are shut down.
My account has been suspended twice, in the early Days when I was getting some help from @coinBuster and @BenBurst with uploads. They reinstated the account without any question, so I do not know if it was just that the account was new, or that uploads were about 1TB a Day, if it was the Mining or if it was that the account was being accessed from multiple IP's?
So for the last few Weeks I have been going it alone Uploading and Mining from a single PC / IP and have also uploaded a load of Photo's :-) However I think the risk is real, but only time will tell?
Ok that's enough from me for now, this is in danger of becoming a monologue....
@RichBC Yeah upload to there different files will help you disguise your account for the use you are giving it lol but they will catch your account when a lot of people is doing this and will shutdown a lot of accounts... I think that if there is a viable way of mining in the cloud is through Amazon EC2 or some simillar tehnology to make it fast enough, but from the calcs i made until now it's not viable YET!
@gpedro Yes agreed however I feel it's one of these thing that you need to persevere with and experiment. It's all to easy to come up with a list of problems and right it off as an idea. Internet related stuff is advancing at a tremendous rate so not being possible Today does not mean it won't be Tomorrow.
However even 5TB which I will soon have virtually for free / no incremental cost would have cost $150 for the Hard Drive.
I have tried mining from a free EC2 instance and that does speed things things up. I have also tried plotting however you cannot do it from a Free Instance as there is insufficient RAM & Hard Disk. A paid instance is just too expensive even if you are careful with spot pricing and then just as you think you have cracked it you get the additional bill for the Data upload. :-(
Here's my setup:
Sony Vaio laptop with i7-4500U @ 1.8GHz
8 GB RAM
4TB Seagate USB powered HDD
AIO Wallet 0.3.8 rc4 - which is kick-ass. no errors and it uses xplotter 0.7. I think these optimized plots will be key on ACD.
Netdrive2 - full version. This is used for mining only.
ACD windows upload software. Using this to upload 25GB plots. Thinking about alternating the size of plots +- 10%.
I'm currently plotting my 4TB HDD (in 1TB increments), so my ACD work is on hold. My laptop cant handle simultaneous plotting. I'll plot the 25GB files on my C drive about 4 at a time, then upload to ACD during the work day and hopefully come home to uploaded files.
I'll also reread the posts that Rich mentioned above. I had read them in the past, but always find new tidbits as I gain knowledge.
i've been working on this for months with no real avail . Ill be watching closely hopefully we can figure it out! i'd love to add an extra 100tb to my portfolio
I have just started a new Free EC2 instance to check what mining speed I get. A couple of Blocks have gone by and with my 3.4TB am seeing around 20MB/s read, so the plot goes through in around 50 Seconds.
There is probably some optimisation of the way the plots are stored in folders that might help a little and it's about 3 times the speed of my own (In the UK) broadband connection. however it's not going to be good enough for a decent sized plot...
Am not going to leave it running at the moment I still have concerns about accessing Amazon Drive from multiple IP?
So as Rich stated, the Amazon upload software is bunk.
Netdrive2 software seems to be working. I've been uploading a 25GB plot for hours now at about 1.2MB/s. Dreadfully slow.
Is there a way to see the status of the upload in Netdrive2? I have task manager open just to see the network activity. That's the only way I can see if its still working.
In other news, I downloaded rclone. That's beyond my pay grade. It might be easy for some, but I'm not that skilled. GUI or bust.
@miraj I also just monitor network activity with NetDrive2. rclone worth persevering with despite being command line. Instructions are quite good and so far has been the most reliable for me.
Have been experimenting with renting a VPS with a decent internet connection. Definitely much faster uploading, but is likely to work out too expensive because of the spec of system you need to Plot and Optimise before you can do the upload.
I am in the process of constructing a Batch File that does the Plot, Optimise and then upload to Amazon. Then deletes the files increments the start nonce & runs again. Still testing but I think the complete cycle for 50GB is going to be about 2Hr 40mins.
However it may all be to no avail as have just had the account locked again. Phone call and it was immediately unlocked, no explanation as to why it was locked, but probably not a coincidence that have been uploading at faster rates for the last 24 Hrs.
Rich, I've installed rclone, and I must say, it's pretty sweet. Even a dummy like me has gotten most of the way there. I have a question. I can't copy a file from my c drive to a folder already created on the ACD. I keep getting "409 conflict" error.
Here's my code:
rclone copy C:\burst\upload\3051863391608293710_15220161_102336_102336 acd:b1
I have to use b1, because burst\plots is already used on ACD. Is there a way to copy the file right to acd:burst\plots (acd being the name I used for the amazon cloud drive)? I don't want to have to create a new folder name for every upload.
Once this works, I'll venture into straight up moving files and start making 50GB plots instead of 25GB....and then get shut down by Amazon.
@miraj Not sure on the error but you can copy to any folder on ACD that you want.
Does folder b1 exist on ACD?
Run rclone config again and just to check that ACD is there?
You can just give rclone the path to the directory with the plots, no file name, and it will copy all the plots.
-v after rclone on the command line gives some verbose output which can be handy
--Transfers -2 on the command line copies 2 files at once if you bandwidth is not maxed with one file. probably not a problem :-)
Thanks for your thoughts, Rich.
I try to copy a plot file to the ACD\burst\plots folder but it says the folder already exists. That is correct, it does already exist on the cloud drive. I just want to put a new file into the folder. rclone doesn't let me so I had to create a new folder called b1. Then when the file is uploaded, I have to move it to burst\plots. Is this the correct code to copy a file to ACD?
rclone copy C:\burst\upload\3051863391608293710_15220161_102336_102336 acd:burst\plots
(I called my amazon cloud drive - acd)
@miraj Strange all of my copies to ACD have been to existing folders without that message?
Got it. Seems the ACD is very case sensitive. My Burst folder has capital "B". Working great now. Just have to wait 8 hours till its finished!
@miraj OK great. Also if you include a directory in the ACD path it will create it.
Be careful around 50GB there is max file size limit. I have been ok with 48GB & 48.8GB but have also found, at least on slow transfers, that failures grow significantly above 25GB
@RichBC Is there code in rclone to upload multiple files, but only upload the next file after the previous one is finished? I'm trying to create a bunch of plot files, then when I'm at work, let this run and it uploads all of the files one at a time. I've noticed that my upload connection drops about once every 3 hours or so. It has to keep restarting the uploads. If that happens, i'd only like to lose the file currently being uploaded.
@miraj Yes you need to put --transfers 1 on the command line.
I think the default is 4. From my home connection I use 1, from the VPS I am using 2 or 3 as it then maxes out the Upload bandwidth that is available.
@RichBC That's exactly what I needed!
Quick update on how the Cloud Mining is going.
To speed up the upload process as an experiment I have rented a VPS for a Month. After some playing I have settled on a minimum spec of 6 Cores, 5GB RAM and 150GB of Hard Disk.
I have written a Batch file that plots 3 x 25GB plots then optimises them and uploads the 3 plots simultaneously to max out the available bandwidth. This process enables me to upload 1TB every 2 Days. There is about a 10% failure rate on the uploads. I am now up to 9TB.
So this will work out at about $2.50 / TB which is just about acceptable. I have only been mining the Plots sparingly as I think it might be the Mining process that get's the Drive shut down? I have however been Mining for the last 24 Hours without problem.
The big issue when a decent amount of plots are up will be the read speed. With a Download Bandwidth of 8MB/s the 9TB takes 4 Minutes to read. :-( I have also been testing Mining from an EC2 Free Instance and get at least twice that speed, but still not really fast enough.
On the faster speeds I have also seen another problem which is a "error ReadFiles" Sometimes it just a couple of errors but on other occasions you get a complete never ending string of errors. Need to understand this and find a solution otherwise it could be a showstopper. perhaps @blago might have some ideas?
If I had confidence that Amazon would not shut down the Drive I think this would have the makings of a great low cost, high value Mining Asset as the cost of creating the plots is only $2.50 /TB and the Annual storage costs only an additional $50 for up to 100TB.