B4A Class AMAZON WEB SERVICES S3 V4 Signature Calculator

The attached zip contains a code module that calculates V4 signatures for Amazon Web Services (AWS) Simple Storage Service (S3) REST API requests, enabling creation of AWS S3 PUT/GET/DELETE requests.

You basically just plug in the properties necessary for a signature calculation and call Signature.

The accompanying example code has 4 worked examples that actually generate the signatures the AWS documentation says they should.

There is extensive in-line documentation.

I could never get:

https://www.b4x.com/android/forum/threads/amazon-s3-library.38699/#post-335436

working but am indebted to it for some hints in a couple of areas.

Enjoy...

EDIT

New version of zip attached (AWS_S3_1_1.zip):

  • added AWS_URI_Is_Path_Style and AWS_End_Point properties
  • replaced AWS_S3_Header_map with AWS_S3_OtherHeader_map (now automatically builds default headers)
  • added URI call to automatically generate http job URI
  • added Authorization call to automatically generate full http job Authorization string
  • added FullHeaderMap call to generate full header map for input to http job
  • changed AWS_S3_Payload to a byte array instead of string
  • changed HexSHA256Hash call's input to a byte array instead of string
 

Attachments

  • AWS_S3.zip
    12.1 KB · Views: 154
  • AWS_S3_1_1.zip
    12.9 KB · Views: 186
Last edited:

JackKirk

Well-Known Member
Licensed User
Longtime User
LIST AWS S3 BUCKET CONTENTS

Attached zip contains a simple practical use of AWS_S3_1_1.zip of previous post.

Change the following statements:

AWS_S3.AWS_Access_Key_ID = "yourAccess_Key_ID"
AWS_S3.AWS_Secret_Access_Key = "yourSecret_Access_Key"
AWS_S3.AWS_Region = "yourRegion"
AWS_S3.AWS_End_Point = "yourEnd_Point"
AWS_S3.AWS_S3_Bucket_Name = "yourBucket_Name"

and run it and you should generate a list of the contents of the appropriate AWS S3 bucket in JSON format.

This is based on:

http://docs.aws.amazon.com/AmazonS3/latest/API/s3-api.pdf
page 98
Example 1:Listing Keys

This document is about 500 pages of detailed examples of everything you can do with AWS S3 - take an example, massage it into the skeleton supplied here and you have AWS S3 at your mercy.:):):)

EDIT

Zip has been updated with a small change in HttpJob class to facilitate AWS S3 file deletes (see post #5)
 

Attachments

  • AWS_S3_Use1.zip
    17.2 KB · Views: 161
Last edited:

JackKirk

Well-Known Member
Licensed User
Longtime User
COPY B4A => AWS S3

If you replace the Main procedure in post #2 with the following code then make the same statement changes as per post #2 you can copy files B4A => AWS S3:
B4X:
#Region  Project Attributes
    #ApplicationLabel: AWS_S3_Use2
    #VersionCode: 1
    #VersionName:
    'SupportedOrientations possible values: unspecified, landscape or portrait.
    #SupportedOrientations: unspecified
    #CanInstallToExternalStorage: False
#End Region

#Region  Activity Attributes
    #FullScreen: False
    #IncludeTitle: True
#End Region

Sub Process_Globals

End Sub

Sub Globals

    Private AWS_job As HttpJob

End Sub

Sub Activity_Create(FirstTime As Boolean)

End Sub

Sub Activity_Resume

    'Set time zone to GMT - all date/times to AWS need to be wrt GMT
    DateTime.SetTimeZone(0)

    'http://docs.aws.amazon.com/AmazonS3/latest/API/s3-api.pdf
    'page 368
    'Example 1: Upload an Object
    '
    'PUT /test1.jpg HTTP/1.1
    'Host: <bucket>.<s3-aws-region>.amazonaws.com
    'x-amz-date: <date as yyyyMMdd'T'HHmmss'Z'>
    'Content-Type: text/plain                      <<<<<<<<<<<<WRONG
    'Content-Length: <length>
    'x-amz-meta-author: <name>
    'Expect: 100-continue
    'Authorization: authorization string
    '[<length> bytes of object data]
    '
    'NOTE: after considerable fustrating fiddling it appears that only
    'Content-Type setting that works is:
    'Content-Type: application/x-www-form-urlencoded
    '
    'AWS_S3_File_Name      : "test1.jpg"
    'AWS_S3_HttpMethod     : "PUT"
    'AWS_S3_Query_map      : Null
    'AWS_S3_Payload        : bytes of File.DirAssets,"test1.jpg"
    'AWS_S3_OtherHeader_map: "Content-Type", "application/x-www-form-urlencoded"
    '                      : "Content-Length", "<length>"
    '                      : "x-amz-meta-author", "<name>"
    '                      : "Expect", "100-continue"

    '   AWS_URI_Is_Path_Style
    '       True = URI is "path style", e.g.:
    '              http://<endpoint>/<bucketname>/<filename>
    '       False = URI is "virtual hosted style", e.g.:
    '               http://<bucketname>.<endpoint>/<filename>
    '    AWS_Access_Key_ID
    '    AWS_Secret_Access_Key
    '    AWS_Region
    '       where S3 bucket resides, as per:
    '       http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
    '       e.g.:
    '           ap-southeast-2
    '    AWS_End_Point
    '       where S3 bucket can be accessed, as per:
    '       http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
    '       e.g.:
    '           s3-ap-southeast-2.amazonaws.com
    '    AWS_S3_Bucket_Name
    '    AWS_S3_File_Name
    '       full name of file in bucket, including any pseudo folders, e.g.:
    '           examplefile.txt
    '           pseudofolder1/pseudofolder2/examplefile.txt
    '    AWS_S3_HttpMethod
    '       "GET", "PUT" or "DELETE"
    '    AWS_S3_Query_map
    '       map of URI query parameter key, value pairs
    '       e.g:
    '            URI is:
    '                http://s3.amazonaws.com/examplebucket?prefix=somePrefix&marker=someMarker&Max-keys=20
    '            URI query parameter key, value pairs:
    '                "prefix", "somePrefix"
    '                "marker", "someMarker"
    '                "max-keys", "20"
    '       e.g:
    '            URI is:
    '                http://s3.amazonaws.com/examplebucket?acl
    '            URI query parameter key, value pairs:
    '                "acl", ""
    '       e.g:
    '            URI does not include a "?"
    '                AWS_S3_Query_map is Null
    '    AWS_S3_Payload
    '    AWS_S3_GMT_DateTime
    '    AWS_S3_OtherHeader_map
    '       map of other request header key, value pairs - exclude following
    '       headers which are added automatically:
    '             host
    '             x-amz-content-sha256
    '             x-amz-date
    '       following headers must be added to this map:
    '             content-type header - if present in request
    '             any other x-amz-* headers
    '       consider including all other headers in order to prevent data tampering

    AWS_S3.AWS_Access_Key_ID = "yourAccess_Key_ID"
    AWS_S3.AWS_Secret_Access_Key = "yourSecret_Access_Key"
    AWS_S3.AWS_Region = "yourRegion"
    AWS_S3.AWS_End_Point = "yourEnd_Point"
    AWS_S3.AWS_S3_Bucket_Name = "yourBucket_Name"
    AWS_S3.AWS_S3_File_Name = "test1.jpg"
    AWS_S3.AWS_S3_HttpMethod = "PUT"

    AWS_S3.AWS_S3_Query_map.Initialize
    AWS_S3.AWS_S3_Query_map.Clear

    Private wrk_in As InputStream
    wrk_in = File.OpenInput(File.DirAssets,"test1.jpg")
    Private wrk_out As OutputStream
    wrk_out.InitializeToBytesArray(1000)
    File.Copy2(wrk_in, wrk_out)
    wrk_in.Close
    wrk_out.Close
    AWS_S3.AWS_S3_Payload = wrk_out.ToBytesArray

    AWS_S3.AWS_S3_GMT_DateTime = DateTime.Now

    AWS_S3.AWS_S3_OtherHeader_map.Initialize
    AWS_S3.AWS_S3_OtherHeader_map.Clear
    AWS_S3.AWS_S3_OtherHeader_map.Put("Content-Type", "application/x-www-form-urlencoded")
    AWS_S3.AWS_S3_OtherHeader_map.Put("Content-Length", NumberFormat2(AWS_S3.AWS_S3_Payload.Length, 1, 0, 0, False))
    AWS_S3.AWS_S3_OtherHeader_map.Put("x-amz-meta-author", "Jack")
    AWS_S3.AWS_S3_OtherHeader_map.Put("Expect", "100-continue")

    Log(AWS_S3.URI)

    'Set up httpjob
    AWS_job.Initialize("AWS_job", Me)
    AWS_job.PutBytes(AWS_S3.URI, AWS_S3.AWS_S3_Payload)

    'Retrieve full header map
    Private wk_hdr_map As Map
    wk_hdr_map.Initialize
    wk_hdr_map = AWS_S3.FullHeaderMap

    Private wrk_ptr As Int

    'For each key, value pair of full header map...
    For wrk_ptr = 0 To wk_hdr_map.Size - 1
   
        'If not host header...
        If wk_hdr_map.GetKeyAt(wrk_ptr) <> "host" Then
       
            'Add it to httpjob
            AWS_job.GetRequest.SetHeader(wk_hdr_map.GetKeyAt(wrk_ptr), wk_hdr_map.GetValueAt(wrk_ptr))

            Log(wk_hdr_map.GetKeyAt(wrk_ptr) & ", " & wk_hdr_map.GetValueAt(wrk_ptr))

        End If

    Next

    Log(AWS_S3.Authorization)

    'Add Authorization header to httpjob
    AWS_job.GetRequest.SetHeader("Authorization", AWS_S3.Authorization)

End Sub

Sub Activity_Pause (UserClosed As Boolean)

End Sub

Sub JobDone(Job As HttpJob)

    Log("JobName = " & Job.JobName & ", Success = " & Job.Success)

    If Job.Success = True Then

        Select Job.JobName

            Case "AWS_job"
           
                Log("test1.jpg uploaded")
           
        End Select
   
    Else
   
        Log("Error: " & Job.ErrorMessage)
   
    End If

    Job.Release

    Activity.Finish

End Sub
 
Last edited:

JackKirk

Well-Known Member
Licensed User
Longtime User
COPY AWS S3 => B4A

If you replace the Main procedure in post #2 with the following code then make the same statement changes as per post #2 you can copy files AWS S3 => B4A:
B4X:
#Region  Project Attributes
    #ApplicationLabel: AWS_S3_Use3
    #VersionCode: 1
    #VersionName:
    'SupportedOrientations possible values: unspecified, landscape or portrait.
    #SupportedOrientations: unspecified
    #CanInstallToExternalStorage: False
#End Region

#Region  Activity Attributes
    #FullScreen: False
    #IncludeTitle: True
#End Region

Sub Process_Globals
   
End Sub

Sub Globals
   
    Private AWS_job As HttpJob

End Sub

Sub Activity_Create(FirstTime As Boolean)

End Sub

Sub Activity_Resume

    'Set time zone to GMT - all date/times to AWS need to be wrt GMT
    DateTime.SetTimeZone(0)
   
    'http://docs.aws.amazon.com/AmazonS3/latest/API/s3-api.pdf
    'page 317
    'Sample Request
    '
    'GET /test1.jpg HTTP/1.1
    'Host: <bucket>.<s3-aws-region>.amazonaws.com
    'x-amz-date: <date as yyyyMMdd'T'HHmmss'Z'>
    'Authorization: authorization string
    '
    'AWS_S3_File_Name      : "test1.jpg"
    'AWS_S3_HttpMethod     : "GET"
    'AWS_S3_Query_map      : Null
    'AWS_S3_Payload        : Null
    'AWS_S3_OtherHeader_map: Null
   
    '   AWS_URI_Is_Path_Style
    '       True = URI is "path style", e.g.:
    '              http://<endpoint>/<bucketname>/<filename>
    '       False = URI is "virtual hosted style", e.g.:
    '               http://<bucketname>.<endpoint>/<filename>
    '    AWS_Access_Key_ID
    '    AWS_Secret_Access_Key
    '    AWS_Region
    '       where S3 bucket resides, as per:
    '       http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
    '       e.g.:
    '           ap-southeast-2
    '    AWS_End_Point
    '       where S3 bucket can be accessed, as per:
    '       http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
    '       e.g.:
    '           s3-ap-southeast-2.amazonaws.com
    '    AWS_S3_Bucket_Name
    '    AWS_S3_File_Name
    '       full name of file in bucket, including any pseudo folders, e.g.:
    '           examplefile.txt
    '           pseudofolder1/pseudofolder2/examplefile.txt
    '    AWS_S3_HttpMethod
    '       "GET", "PUT" or "DELETE"
    '    AWS_S3_Query_map
    '       map of URI query parameter key, value pairs
    '       e.g:
    '            URI is:
    '                http://s3.amazonaws.com/examplebucket?prefix=somePrefix&marker=someMarker&Max-keys=20
    '            URI query parameter key, value pairs:
    '                "prefix", "somePrefix"
    '                "marker", "someMarker"
    '                "max-keys", "20"
    '       e.g:
    '            URI is:
    '                http://s3.amazonaws.com/examplebucket?acl
    '            URI query parameter key, value pairs:
    '                "acl", ""
    '       e.g:
    '            URI does not include a "?"
    '                AWS_S3_Query_map is Null
    '    AWS_S3_Payload
    '    AWS_S3_GMT_DateTime
    '    AWS_S3_OtherHeader_map
    '       map of other request header key, value pairs - exclude following
    '       headers which are added automatically:
    '             host
    '             x-amz-content-sha256
    '             x-amz-date
    '       following headers must be added to this map:
    '             content-type header - if present in request
    '             any other x-amz-* headers
    '       consider including all other headers in order to prevent data tampering
   
    AWS_S3.AWS_Access_Key_ID = "yourAccess_Key_ID"
    AWS_S3.AWS_Secret_Access_Key = "yourSecret_Access_Key"
    AWS_S3.AWS_Region = "yourRegion"
    AWS_S3.AWS_End_Point = "yourEnd_Point"
    AWS_S3.AWS_S3_Bucket_Name = "yourBucket_Name"
    AWS_S3.AWS_S3_File_Name = "test1.jpg"
    AWS_S3.AWS_S3_HttpMethod = "GET"

    AWS_S3.AWS_S3_Query_map.Initialize
    AWS_S3.AWS_S3_Query_map.Clear

    Private null_bytes() As Byte
    AWS_S3.AWS_S3_Payload = null_bytes

    AWS_S3.AWS_S3_GMT_DateTime = DateTime.Now

    AWS_S3.AWS_S3_OtherHeader_map.Initialize
    AWS_S3.AWS_S3_OtherHeader_map.Clear

    Log(AWS_S3.URI)
   
    'Set up httpjob
    AWS_job.Initialize("AWS_job", Me)
    AWS_job.Download(AWS_S3.URI)
   
    'Retrieve full header map
    Private wk_hdr_map As Map
    wk_hdr_map.Initialize
    wk_hdr_map = AWS_S3.FullHeaderMap

    Private wrk_ptr As Int

    'For each key, value pair of full header map...
    For wrk_ptr = 0 To wk_hdr_map.Size - 1
       
        'If not host header...
        If wk_hdr_map.GetKeyAt(wrk_ptr) <> "host" Then
           
            'Add it to httpjob
            AWS_job.GetRequest.SetHeader(wk_hdr_map.GetKeyAt(wrk_ptr), wk_hdr_map.GetValueAt(wrk_ptr))

            Log(wk_hdr_map.GetKeyAt(wrk_ptr) & ", " & wk_hdr_map.GetValueAt(wrk_ptr))

        End If

    Next
   
    Log(AWS_S3.Authorization)
   
    'Add Authorization header to httpjob
    AWS_job.GetRequest.SetHeader("Authorization", AWS_S3.Authorization)
   
End Sub

Sub Activity_Pause (UserClosed As Boolean)

End Sub

Sub JobDone(Job As HttpJob)
   
    Log("JobName = " & Job.JobName & ", Success = " & Job.Success)
   
    If Job.Success = True Then
   
        Select Job.JobName
   
            Case "AWS_job"

                Private wrk_in As InputStream
                wrk_in = Job.GetInputStream
                Private wrk_out As OutputStream
                wrk_out = File.OpenOutput(File.DirRootExternal, "test1.jpg", False)
                File.Copy2(wrk_in, wrk_out)
                wrk_in.Close
                wrk_out.Close
               
                Log("test1.jpg downloaded")
               
        End Select
       
    Else
       
        Log("Error: " & Job.ErrorMessage)
       
    End If
   
    Job.Release
   
    Activity.Finish
   
End Sub
 

JackKirk

Well-Known Member
Licensed User
Longtime User
DELETE AWS S3

If you replace the Main procedure in post #2 with the following code then make the same statement changes as per post #2 you can delete files in AWS S3:
B4X:
#Region  Project Attributes
    #ApplicationLabel: AWS_S3_Use4
    #VersionCode: 1
    #VersionName:
    'SupportedOrientations possible values: unspecified, landscape or portrait.
    #SupportedOrientations: unspecified
    #CanInstallToExternalStorage: False
#End Region

#Region  Activity Attributes
    #FullScreen: False
    #IncludeTitle: True
#End Region

Sub Process_Globals
   
End Sub

Sub Globals
   
    Private AWS_job As HttpJob

End Sub

Sub Activity_Create(FirstTime As Boolean)

End Sub

Sub Activity_Resume

    'Set time zone to GMT - all date/times to AWS need to be wrt GMT
    DateTime.SetTimeZone(0)
   
    'http://docs.aws.amazon.com/AmazonS3/latest/API/s3-api.pdf
    'page 305
    'Sample Request
    '
    'DELETE /test1.jpg HTTP/1.1
    'Host: <bucket>.<s3-aws-region>.amazonaws.com
    'x-amz-date: <date as yyyyMMdd'T'HHmmss'Z'>
    'Content-Type: text/plain
    'Authorization: authorization string
    '
    'AWS_S3_File_Name      : "test1.jpg"
    'AWS_S3_HttpMethod     : "DELETE"
    'AWS_S3_Query_map      : Null
    'AWS_S3_Payload        : Null
    'AWS_S3_OtherHeader_map: "Content-Type", "text/plain"
   
    '   AWS_URI_Is_Path_Style
    '       True = URI is "path style", e.g.:
    '              http://<endpoint>/<bucketname>/<filename>
    '       False = URI is "virtual hosted style", e.g.:
    '               http://<bucketname>.<endpoint>/<filename>
    '    AWS_Access_Key_ID
    '    AWS_Secret_Access_Key
    '    AWS_Region
    '       where S3 bucket resides, as per:
    '       http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
    '       e.g.:
    '           ap-southeast-2
    '    AWS_End_Point
    '       where S3 bucket can be accessed, as per:
    '       http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
    '       e.g.:
    '           s3-ap-southeast-2.amazonaws.com
    '    AWS_S3_Bucket_Name
    '    AWS_S3_File_Name
    '       full name of file in bucket, including any pseudo folders, e.g.:
    '           examplefile.txt
    '           pseudofolder1/pseudofolder2/examplefile.txt
    '    AWS_S3_HttpMethod
    '       "GET", "PUT" or "DELETE"
    '    AWS_S3_Query_map
    '       map of URI query parameter key, value pairs
    '       e.g:
    '            URI is:
    '                http://s3.amazonaws.com/examplebucket?prefix=somePrefix&marker=someMarker&Max-keys=20
    '            URI query parameter key, value pairs:
    '                "prefix", "somePrefix"
    '                "marker", "someMarker"
    '                "max-keys", "20"
    '       e.g:
    '            URI is:
    '                http://s3.amazonaws.com/examplebucket?acl
    '            URI query parameter key, value pairs:
    '                "acl", ""
    '       e.g:
    '            URI does not include a "?"
    '                AWS_S3_Query_map is Null
    '    AWS_S3_Payload
    '    AWS_S3_GMT_DateTime
    '    AWS_S3_OtherHeader_map
    '       map of other request header key, value pairs - exclude following
    '       headers which are added automatically:
    '             host
    '             x-amz-content-sha256
    '             x-amz-date
    '       following headers must be added to this map:
    '             content-type header - if present in request
    '             any other x-amz-* headers
    '       consider including all other headers in order to prevent data tampering
   
    AWS_S3.AWS_URI_Is_Path_Style = True
    AWS_S3.AWS_Access_Key_ID = "yourAccess_Key_ID"
    AWS_S3.AWS_Secret_Access_Key = "yourSecret_Access_Key"
    AWS_S3.AWS_Region = "yourRegion"
    AWS_S3.AWS_End_Point = "yourEnd_Point"
    AWS_S3.AWS_S3_Bucket_Name = "yourBucket_Name"
    AWS_S3.AWS_S3_File_Name = "test1.jpg"
    AWS_S3.AWS_S3_HttpMethod = "DELETE"

    AWS_S3.AWS_S3_Query_map.Initialize
    AWS_S3.AWS_S3_Query_map.Clear

    Private null_bytes() As Byte
    AWS_S3.AWS_S3_Payload = null_bytes

    AWS_S3.AWS_S3_GMT_DateTime = DateTime.Now

    AWS_S3.AWS_S3_OtherHeader_map.Initialize
    AWS_S3.AWS_S3_OtherHeader_map.Clear
    AWS_S3.AWS_S3_OtherHeader_map.Put("Content-Type", "text/plain")

    Log(AWS_S3.URI)
   
    'Set up httpjob
    AWS_job.Initialize("AWS_job", Me)
    AWS_job.DeleteRequest(AWS_S3.URI)
   
    'Retrieve full header map
    Private wk_hdr_map As Map
    wk_hdr_map.Initialize
    wk_hdr_map = AWS_S3.FullHeaderMap

    Private wrk_ptr As Int

    'For each key, value pair of full header map...
    For wrk_ptr = 0 To wk_hdr_map.Size - 1
       
        'If not host header...
        If wk_hdr_map.GetKeyAt(wrk_ptr) <> "host" Then
           
            'Add it to httpjob
            AWS_job.GetRequest.SetHeader(wk_hdr_map.GetKeyAt(wrk_ptr), wk_hdr_map.GetValueAt(wrk_ptr))

            Log(wk_hdr_map.GetKeyAt(wrk_ptr) & ", " & wk_hdr_map.GetValueAt(wrk_ptr))

        End If

    Next
   
    Log(AWS_S3.Authorization)
   
    'Add Authorization header to httpjob
    AWS_job.GetRequest.SetHeader("Authorization", AWS_S3.Authorization)
   
End Sub

Sub Activity_Pause (UserClosed As Boolean)

End Sub

Sub JobDone(Job As HttpJob)
   
    Log("JobName = " & Job.JobName & ", Success = " & Job.Success)
   
    If Job.Success = True Then
   
        Select Job.JobName
   
            Case "AWS_job"

                Log(Job.GetString)
               
                Log("test1.jpg deleted")
               
        End Select
       
    Else
       
        Log("Error: " & Job.ErrorMessage)
       
    End If
   
    Job.Release
   
    Activity.Finish
   
End Sub
 

DonManfred

Expert
Licensed User
Longtime User
You should make a class out of them ALL and release this Class :)
 

JackKirk

Well-Known Member
Licensed User
Longtime User
You should make a class out of them ALL and release this Class
Hi, these are basically a result of some research for a major project I seem to have got sucked into - I guess they are more in the vein of a tutorial than a solution - although I don't consider myself qualified to be tutoring anyone in this forum.

One of the issues with coming up with a full blown class is that there are literally 100's of ways of manipulating AWS S3 buckets and their contents - see:

http://docs.aws.amazon.com/AmazonS3/latest/API/s3-api.pdf

it is 500 pages!

Maybe when I get the project under control I could have a go - I will probably be doing some form of coagulation in that anyrate.

Thanks for your interest...
 
Last edited:

JackKirk

Well-Known Member
Licensed User
Longtime User
The real core of this is the signature calculation which is only indirectly involved with Http.

However, making practical use of the signature calculation involves Http.

The only possible roadblock I can think of would be availability of the libraries relied upon by the signature calculation.

Eventually my project is going to need a B4J implementation - I would be very interested in knowing how you go Rubén.
 
Last edited:
Top