S3 Functions in Python using Boto

 1. Copy from one folder in a bucket to another using Boto. Configure Boto AWS as mentioned in the documentation here

import boto3
import os,sys
bucketName="YOUR BUCKET NAME"

# set this to True if you want the objects in copyToPathDirectory to be deleted first
# set to False if the object being copied has the same name
deleteObjectsFirst = False

# Assuming Boto is setup
s3 = boto3.resource('s3')
s3Client = boto3.client('s3')
bucket = s3.Bucket(bucketName) 
copyToPathDirectory = "the/directory/youwant/to/copyto/"

copyFromPathNames = [
  "path1/to/copyfrom/",
  "path2/to/copyfrom/",
  "and/so/on/"
]

# WARNING: This will delete all objects in your folder
# Please check your path before running
if deleteObjectsFirst:
  delete_key_list = []
  for copyFromPath in copyFromPathNames:
    for obj in bucket.objects.filter(Prefix = copyFromPath):
      deleteKey = copyToPathDirectory + obj.key
      delete_key_list.append({"Key":deleteKey})
      if len(delete_key_list) > 100:
          s3Client.delete_objects(Bucket=bucketName, Delete={"Objects":delete_key_list})
          delete_key_list = []

  if len(delete_key_list) > 0:
      s3Client.delete_objects(Bucket=bucketName, Delete={"Objects":delete_key_list})

# Copy from one folder to another
for copyFromPath in copyFromPathNames:
  for obj in bucket.objects.filter(Prefix = copyFromPath):
        copy_source = {'Bucket': bucketName,'Key': obj.key}
        toKey = copyToPathDirectory + obj.key
        s3.meta.client.copy(copy_source, bucketName, toKey)

2. Download files from S3 to local directory

import boto3
import os,sys

bucketName="YOUR BUCKET NAME"
remoteDirectoryName="YOUR DIRECTORY NAME"

s3_resource = boto3.resource('s3')
bucket = s3_resource.Bucket(bucketName) 
for obj in bucket.objects.filter(Prefix = remoteDirectoryName):
    if not os.path.exists(os.path.dirname(obj.key)):
        os.makedirs(os.path.dirname(obj.key))
    bucket.download_file(obj.key, obj.key)

Comments

Post a Comment

Popular posts from this blog

Authoritative Server - MMO

Latency, Authoritative Server, Lag Compensation, Course Correction, Packet Drops and Client Side Prediction. These are things you would worry about when attempting to build a MMO game. For companies where the delay matters these logics are a must. This post is on setting up an Authoritative Server. There are different ways of setting up the authoritative server. Some of them is mentioned here. You have N clients and one client acts as the authoritative server: Imagine N clients with different machine configurations. The server would need to decide the best possible machine and have the physics simulation run in that. We then have issues with client disconnecting in which case we need to swap the authoritative client to another machine. This means we have to maintain all the state the game was in and move it to the new authoritative client. In an online environment this is simple nothing but nonsense. There is a dedicated server which simulates everything including physics and ...
Read more

Code dependencies

Are you struggling to make a given set of scripts from a project compile error free? If that is true, then you are in the same boat as I am right now. I have come across projects where logic is scattered around to get a functionality out of the pipeline with limited value to object oriented programming, and the most valuable and time saving of all is the plugin architecture. This is as best as it can get. Let me explain.. A producer has an idea and wants to make a quick prototype. He opens a new project, drags and drops the packages he needs and boom the game runs. Obviously you don't expect a producer to code (unless he is from development). The game would not be perfect, but at least it would get the ball rolling and you don't need a developer to assist the producer. Now, to do this, you would want the packages to follow a common rule. You can do this using an interface and have classes implement them. For example, the avatar package and the pet package can implement the...
Read more