Mark Richardson

Mark Richardson

I am a Profesional Services Engineer specializing in backup, storage, and automation

Using the AWS CLI to prepare for Commvault S3 Cloud Library

6 minute read

To create an AWS S3 based Cloud Library in Commvault a number of steps need to be completed in AWS prior. These steps depend on what you are trying to do in Commvault. We will focus on a basic AWS S3 Cloud Library and show how to configure AWS for both a MediaAgent on-premise and an EC2 MediaAgent.

Now, of course we could do all of this via the AWS Management Console which is great too but…not as fun as the AWS CLI. So with that being said we will do everything with the aws cli and, in my case, on a Mac. Any Linux/Unix system will do. If you are using Windows then I recommend PowerShell and I believe the commands are very similar if not exactly the same.

How to create an AWS Cloud Lib for Commvault if MediaAgent is On-Premises

The goal of this is to create everything to prepare for the creation of a Cloud Library in Commvault using AWS S3 storage. When done we will have the three (3) pieces of information necessary to create the library (bucket, SecretAccessKey, and AccessKeyId). The bucket isn’t really necessary since the permissions applied will allow for creation of a new bucket when creating the cloud library. Please, note, that the bucket name is globally unique and must be lower case. For more detail on bucket naming restrictions see the following link

  1. If the aws cli is not already installed on your Mac or Linux system then download and install. You can use pip as well.

    Download and uncompress

     curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
 unzip awscli-bundle.zip

    Install

     sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

    Configure profile

     aws configure

  2. If AWS CLI is already installed, verify your running profile. This will help us determine if you have the rights to accomplish the tasks

     aws configure list

  3. Verify the permissions of user associated with running profile

    Verify running user associated with the running profile

     aws sts get-caller-identity

    List groups associated with user found associated to the current running profile from previous command

     aws iam list-groups-for-user --user-name MyUser

    List policies associated with Group(s)

     aws iam list-attached-group-policies --group-name MyGroup

    List permissions associated with Policy(s) shown from output in previous command

     aws iam get-policy-version --policy-arn arn:aws:iam::aws:policy/AdministratorAccess --version-id v1

  4. Assuming permission checks above are good, then create bucket if not already created.

     aws s3 mb s3://mycvtestbucket007943

    List all buckets to verify created. You can pipe to grep if there are a lot.

     aws s3 ls

  5. Create Commvault user account for access. This is basically a service account.

     aws iam create-user --user-name CvLibUser

  6. Setup bucket Access for user. NOTE: The default json provided is fairly open so it can be restricted to the S3 bucket.

     aws iam put-user-policy --user-name CvLibUser  --policy-name CVPolicy --policy-document file://aws_cv.json

    The json for Commvault can be downloaded here or is detailed below.

    
 {
     "Version": "2012-10-17",
     "Statement": [
         {
             "Action": [
                 "s3:ListBucket",
                 "s3:GetBucketLocation",
                 "s3:CreateBucket",
                 "s3:ListAllMyBuckets",
                 "s3:PutObject",
                 "s3:GetObject",
                 "s3:PutObjectTagging",
                 "s3:DeleteObject"
             ],
             "Effect": "Allow",
             "Resource": "*"

         }
     ]
 }

  7. Create keys for the service account user created previously. We can use this to specify in Commvault Cloud Lib Creation. Don’t lose as only created once! Now we have what we need. Go create a Cloud library!

     aws iam create-access-key --user-name CvLibUser > CvLibUser.keys

  8. So we can put this together in a script. See example below.

     #!/bin/sh

 # ./cv-aws-make-lib.sh CVLibUser4 mycvtestbucket00789 file://aws_cv.json
 Username=$1
 Bucketname=$2
 Jsonfile=$3

 aws iam create-user --user-name $Username
 aws iam put-user-policy --user-name $Username --policy-name CVPolicy --policy-document $Jsonfile
 aws s3 mb s3://$Bucketname
 aws iam create-access-key --user-name $Username > $Username.keys

How to create an AWS Cloud Library if MediaAgent is an EC2 Instance

If the MediaAgent to be used is an EC2 Instance than the steps are a bit more involved. Since the MediaAgent is an EC2 Instance we can use the more secure and preferred method of using an IAM service role. These are roles that can be assumed by an AWS service. A role is an entity that has its own set of permissions, but that isn’t a user or group. For EC2, IAM dynamically provides temporary credentials to the EC2 instance, and these credentials are automatically rotated for you.

  1. We will use the default VPC but any will do. We need it’s id.

     aws ec2 describe-vpcs

  2. Setup security group to allow network access. Specify vpc-id from previous step

     aws ec2 create-security-group --group-name CVSecurityGroup --description "Commvault security group" --vpc-id vpc-yourId

  3. Allow inbound ports. Restricted. Cidr specified is for example only and should be limited appropriately

     aws ec2 authorize-security-group-ingress --group-id sg-YourGroupId --protocol tcp --port 22 --cidr 0.0.0.0/0
 aws ec2 authorize-security-group-ingress --group-id sg-YourGroupId --protocol tcp --port 80 --cidr 0.0.0.0/0
 aws ec2 authorize-security-group-ingress --group-id sg-YourGroupId --protocol tcp --port 443 --cidr 0.0.0.0/0
 aws ec2 authorize-security-group-ingress --group-id sg-YourGroupId --protocol tcp --port 8400-8403 --cidr 0.0.0.0/0
 aws ec2 authorize-security-group-ingress --group-id sg-YourGroupId --protocol tcp --port 3389 --cidr 0.0.0.0/0

  4. Allow outbound ports. All open.

     aws ec2 authorize-security-group-egress --group-id sg-YourGroupId --ip-permissions IpProtocol=tcp,FromPort=0,ToPort=65535,IpRanges='[{CidrIp=0.0.0.0/0}]'

  5. Setup key pair for EC2 Instance creation. When you launch an instance, you specify the key pair. You can specify an existing key pair or a new key pair that you create at launch. At boot time, the public key content is placed on the instance in an entry within ~/.ssh/authorized_keys. To log in to your instance, you must specify the private key when you connect to the instance.

     aws ec2 create-key-pair --key-name YourCVKeyPair --query 'KeyMaterial' --output text > YourCVKeyPair.pem

  6. Create a Service Role for access from EC2. You specify the service that needs access in the trust policy. In this case it is EC2.

      aws iam create-role --role-name YourCVRole --assume-role-policy-document file://cv_trust.json

    The json for the trust Policy is below (cv_trust.json):

    
  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Service": [
            "ec2.amazonaws.com"
          ]
        },
        "Action": "sts:AssumeRole"
      }
    ]
  }

  7. Apply Role Policy to Role to attain specific permissions to S3. Just like we provide permissions to a user we do the same for the role. The role now has access to S3 and has the necessary bucket permissions

      aws iam put-role-policy --role-name YourCVRole --policy-name YourCVPolicy --policy-document file://aws_cv.json

    The json for Commvault can be downloaded here.

  8. Create EC2 Instance Profile to attach to EC2 MediaAgent. An instance profile is required and is a container for an IAM role that you can use to pass role information to an EC2 instance when the instance starts

     aws iam create-instance-profile --instance-profile-name YourCVMediaAgent

  9. Associate Role to Instance Profile

     aws iam add-role-to-instance-profile --role-name YourCVRole --instance-profile-name YourCVMediaAgent

  10. List profiles. Once confirmed it is there then we are ready to create our Commvault EC2 Instance and associate the Instance Profile we just created to it.

    aws iam list-instance-profiles

  11. Get the Commvault ami image from the AWS MarketPlace. No need to install as it comes pre-installed! Let’s save time!

    • Login to the AWS Management Console and then go to EC2 > AWS MarketPlace. Search for Commvault
    • Select Commvault MediaAgent - BYOL Edition. Click Continue to Subcscribe. No worries you are not signing up for anything!
    • Confirm Subscription then click configure. Select options and region. Click Continue to Launch. NOTE: Once subscribed you can get the Ami-Id here!
    • In Action Selection box select Copy To Service Catalog. Click Copy To Service Catalog for use later.

  12. Launch Instance of Commvault Image. It is Linux FREL as well, so that’s cool.

     aws ec2 run-instances --image-id ami-YourAmiID --count 1 --instance-type t2.micro --key-name YourCVKeyPair --security-group-ids sg-YourGroupId

  13. Associate Instance Profile to EC2 MediaAgent. Note the json output from previous command to get the instance-id.

    aws ec2 associate-iam-instance-profile --instance-id i-YourInstanceId --iam-instance-profile Name=YourCVMediaAgent

  14. Get the public IP of the EC2 Instance so you can connect to it

    aws ec2 describe-instances --instance-ids i-YourInstanceId

  15. Remote to your EC2 instance. Change security on pem file if not done already to allow ssh access

    chmod go= CVKeyPair.pem
ssh -i CVKeyPair.pem ec2-user@YourHostName.us-west-1.compute.amazonaws.com

  16. Configure the EC Instance with your CommCell

    • run sudo su
    • run cd/opt/commvault_image/UnixCustomPackage/pkg
    • run ./cvpkgadd
    • Interactive UI guides through the installation steps
      • NOTE: If you have a home lab setup and are testing, then you can setup a Commvault one-way firewall with blocked for incoming to your home (CommServe/MA). Set your public IP as CommServe hostname interface for the EC2 Instance.
    • Once install is completed , MediaAgent node gets listed in the registered Commcell and has IAM to access S3 library and is ready for Commvault Cloud Library configuration

You may also enjoy