ADB322 Workshop Guide - Google Docs

Implementing a Flight Simulator Interface Using AI, Virtual Reality and Big Data on AWS

re:INVENT 2017 ABD322 Workshop

STUDENT WORKBOOK Version 1.0.1

1/40



Revision History Version

Author

Date

Comments

1.0.0

teichtah@

20/11/2017 Initial public release

1.0.1

teichtah@

26/11/2017 Updated after further testing

2/40



Introduction Recent technology improvements and ever decreasing price points, have now made Augmented Reality (AR), Virtual Reality (VR) and Simulators, accessible to everyday consumers. As the proliferation of these technologies increasingly pervade into society, managing, understanding and controlling the data they generate becomes increasingly important. This workshop aims to introduce participants to methods and architectures suitable for the instrumentation of AR, VR and simulators. Focusing on streaming data, a serverless approach is taken as a platform for the ingestion, processing and visualization of flight simulator data.

Note: C redits will be supplied for this workshop. There will be no cost to the participant for the consumption of resources for the duration of this workshop. IMPORTANT NOTE: This workshop has been expressly designed to function in the AWS apsoutheast2 (Sydney) region. Please ensure you have selected the Asia Pacific (Sydney) region from the region selection drop down.

Prerequisites 1. AWS CLI Tools Please refer to the installation guide found here for instructions on how to install and configure these tools for Windows, OS X and Linux. 2. A functional AWS account. If you need to create one you can do so h ere. 3. A valid EC2 Key Pair. If this needs to be created, please following the instructions here . Please be sure to download a copy of your private key and retain this in a secure location. This workshop r elies on access to this private key. 4. For windows users an SSH client such as Putty (found here ). OS X and Linux users are able to use the built in SSH client.

CloudFormation Stack Architecture

3/40



End to End Architecture

What can you expect as a result from today’s workshop If you successfully progress through all the steps and learnings of this workshop, you should end up with a fully functional UI representing the streaming data at the end of the processing pipeline. You will be guided through each step of the workshop. Clear instructions and guidance is provided to ensure that each step is as easy as possible to complete. A sample of the expected application is below.

4/40



Activity 1: Preparing your “simulator” source in the cloud Allocated Time: 10 minutes Introduction The source of all flight, weather and time data for this workshop will be generated by an Amazon EC2 (Elastic Cloud Compute) instance. This will be created in your own account and will automatically stream data over MQTT to any MQTT client that connects. Approach A cloudformation template will be used to automate the creation of an AWS VPC (Virtual Private Cloud), and an EC2 instance. You will be using the AWS CLI (AWS Command Line Interface) tools to do this. The AWS CLI is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts. Prerequisites 1. Access to your AWS account and your Key Pair private key (.pem) file for the name of the key pair that you will use to log in to the instance securely. 2. For windows users an SSH client such as Putty (found here ). OS X and Linux users are able to use the built in SSH client. Task 1: Log in to your AWS account 1. Using a web browser of your choice, go to the AWS home page h ere and click the S ign In to the Console b utton.

2.

This will take you to the login page of the A WS Console . Please login with the email address and password you setup when creating your AWS account.

5/40



3.

All activities for this workshop will take place in the AWS Asia Pacific Sydney (apsoutheast2) region. Select the region drop down located on the top right hand corner of the main console and select “A sia Pacific (Sydney) ”. DO NOT CHANGE THIS REGION DURING THE WORKSHOP

Task 2: Prepare to launch the “simulator” and “workstation” CloudFormation stack 1. Once successfully logged in, use the AWS console to search for cloudformation, in the AWS Services search bar.

2.

Selecting C loudFormation from the down will open the CloudFormation console. Using the CloudFormation console, select C reate Stack.

6/40



3.

AWS offers the option to create a stack via a template designer, from a preexisting sample template, or to supply your own via an S3 URL. Select “ Specify an Amazon S3 template URL” and provide the following URL

http://reinventabd322workshop.s3apsoutheast2.amazonaws.com/simulatorDataStack.json

Select “N ext ” 4.

Specifying details about your stack will allow you to provide a S tack Name a nd configure other parameters exposed by the template provided. Use “ simDataProducer ” for the Stack Name . Note this name as you will require it later in the workshop. Do NOT select “Next”

7/40



5.

All Parameters o ther than the K eyPairName should be left untouched. Enter your Key Pair Name in the K eyPairName f ield. You must have access to the private key for this Key Pair if you don’t, please create a new key using the instructions h ere . Select “N ext ”.

6. 7.

There are no additional options to be set on the O ptions screen. Select “N ext ”. Validate that the all the stack details and configuration options are as specified in this Document. If so, select “C reate ” to start the creation of your CloudFormation Stack.

8/40



Note: The K eyPairName parameter for your setup will be different from the one shown above and should show the K eyPairName p rovided in step 5. 8.

Progress of the stack creation can be monitored on the C loudFormation Stacks page, which is provided once the stack creation has commenced. Ensure the stack has a “ CREATE_IN_PROGRESS ” status. If not, please contact a proctor.

9.

The stack should take around 5 minutes to complete, after which, the status of your CloudFormation stack will change from “ CREATE_IN_PROGRESS ” to “ CREATE_COMPLETE ”. At this stage, the stack has been created and is ready for use. Before we can use the stack, we need to obtain a few details about the resources that have been created. These can be read from the O utputs section of the CloudFormation console. 9/40



Four important outputs have been provided. The simDataProducerVPCId (in this case vpc30525954), IPAddress (in this case 13.54.65.8), Subnet (in this case subnet41ac6308) and SecurityGroup (in this case subnet41ac6308). Please note these as they will be required later. 10.

With the simulator stack created, creation of the workstation stack is next. Using the CloudFormation console, select C reate Stack.

11.

AWS offers the option to create a stack via a template designer, from a preexisting sample template, or to supply your own via an S3 URL. Select “ Specify an Amazon S3 template URL” and provide the following URL http://reinventabd322workshop.s3apsoutheast2.amazonaws.com/simulatorWor kstationStack.json

10/40



12.

13.

Specifying details about your stack will allow you to provide a S tack Name a nd configure other parameters exposed by the template provided. Use s imulatorWorkstation f or the Stack Name . Note this name as you will require it later in the workshop. D o NOT select “Next”.

This stack relies on information provided by the stack just created. Specifically, InstanceSecurityGroup . KeyPairName , Subnet, IPAddress and SimulatorIPAddress . Using the information noted in step 9, complete these parameters.

Select “N ext ”. 14. 15.

There are no additional options to be set on the O ptions screen. Select “N ext ”. Validate that the all the stack details and configuration options are as specified in this Document.

11/40



As we are creating an IAM instance role and instance profile during this step, click “I acknowledge that AWS CloudFormation might create IAM resources. ”

16.

Next, select “C reate ” to start the creation of the CloudFormation Stack. Progress of the stack creation can be monitored on the C loudFormation Stacks page, which is provided once the stack creation has commenced. Ensure the stack has a “ CREATE_IN_PROGRESS ” status. You can click the not, please contact a proctor.

button to refresh the status. If

12/40



17.

After some time, the status of your CloudFormation stack will change from “ CREATE_IN_PROGRESS ” to “ CREATE_COMPLETE ”. At this stage, the stack has been created and is ready for use. Before we can use the stack, we need to obtain a few details about the resources that have been created. These can be read from the Outputs section of the CloudFormation console.

A single important output has been provided. The IPAddress (in this case 54.79.63.71). Please note these as they will be required later. Task 3: Validate that the flight data is being streamed by the “simulator” 1. Accessing the workstation created in the previous task, requires access to an SSH client. Linux and OSX users will be able to use the native SSH client. Windows users will be required to use Putty. Follow the directions below for your platform to access your simulator workstation. Linux and OSX NOTE: If required change permissions of your .pem file to be user readable only. Command to change this is: chmod 600 .pem A successful connection will allow you to access your simulator workstation. 13/40



Windows

In the hostname text input, enter the IP address of the simulator workstation. Next, click on SSH → Auth. In the Private key for authentication , click Browse and select your Key Pair private key file.

14/40



2.

Click “O pen ”. MQTT stands for MQ Telemetry Transport. It is a publish/subscribe, extremely simple and lightweight messaging protocol, designed for constrained devices and lowbandwidth, highlatency or unreliable networks. The design principles are to minimise network bandwidth and device resource requirements whilst also attempting to ensure reliability and some degree of assurance of delivery. These principles also turn out to make the protocol ideal of the emerging “machinetomachine” (M2M) or “Internet of Things” world of connected devices, and for mobile applications where bandwidth and battery power are at a premium

Once connected to your simulator workstation enter the following command to validate that the simulator data source is streaming data over MQTT. This uses an open source MQTT client (details can be found here ). It connects to the simulator you created in task 2 on port 1883 and subscribes to the /flight/data topic. When successful, you can expect to see an output as below.

15/40



3.

The SSH session can remain open as it will be used in later activities. However, it is possible that the session may timeout (depending on your client settings). If this does happen, reopen the SSH session as described in step 1.

What has been achieved so far. In this activity the simDataProducer and simDataWorkstation EC2 instances have been created. These instances have been created in the apsoutheast2 (Sydney) region and created within a dedicated VPC with the appropriate permissions, security groups and IAM policies. A MQTT stream has also been created. This stream, pushes raw unprocessed flight data from the simDataProducer.

END OF ACTIVITY 1

16/40



Activity 2: Consuming the simulator data stream

Allocated Time: 20 minutes Introduction Having both the simulator data generator and a workstation created via CloudFormation, a data processing pipeline is required. This pipeline will consume data sent (after processing) from the simulator workstation. This data is then processed through various stages of the pipeline and prepared for consumption. Approach Using the AWS console, an Amazon Kinesis stream will be created. This stream will be later used as a trigger for an AWS Lambda function that is responsible for fanningout the simulator data. Prerequisites 1. Activity 1 and tasks 1,2 and 3 must be completed prior to commencing this activity. Task 1: Create the Amazon Kinesis stream.

1.

Selecting S ervices for the AWS console page, type in K inesis a nd select it from the drop down menu.

2.

The Amazon Kinesis console allows the configuration of Kinesis Streams, Kinesis Firehost and Kinesis Analytics. Click on “G et Started ” then select “C reate Data Stream” .

17/40



3.

The Streams console allows the creation of an Amazon Kinesis stream. In the Create Kinesis stream console, enter “s imFlightData ” as the K inesis stream name

Amazon Kinesis shards are a unit of throughput capacity. Each shard ingests up to 1MB/sec and 1000 records/sec, and emits up to 2MB/sec. A single shard will be used for this workshop.

5.

Select “C reate Kinesis stream ”. It may take several minutes for the Kinesis stream to be created. The status of the creation 18/40



can be tracked on the main Kinesis streams page.

6.

When the status is A CTIVE the Kinesis stream has been created and is ready for use. Like all AWS resources an Amazon Kinesis stream will be allocated a unique ARN (Amazon Resource Name). The ARN for the newly created Amazon Kinesis stream can be found on the Kinesis Streams details page. This ARN will need to be recorded as it required later in this workshop. Click on “s imFlightData ” to access the details page.

The ARN can be found in the S tream ARN section.

Note the ARN as it will be required later Task 2: Stream data from the simulator into Kinesis 1. Using the existing SSH session to the workstation, created in Activity 1, an MQTT client and Amazon Kinesis producer will be executed. This consumer will accept the simulator MQTT stream, process this data, and publish it on to an Amazon Kinesis stream. If you have had to reconnect the SSH session use the commands below once reconnected. 19/40



If you are reusing the SSH session created in activity 1, you will need to issue a Ctrl^C command to terminate the running application. Then enter the commands below. A successful execution will look similar to

At three second intervals, statistics will be produced.

The format for these statistics is # o f data elements / total d ata elements / average elements/second

DO NOT CLOSE THE SSH SESSION. T his will terminate the consumer and no data will flow into Kinesis. If you do happen to accidently terminate the SSH session, please contact a proctor. Task 3: Verify data is flowing into Kinesis 1. Amazon Kinesis offers a full suite of metrics and monitoring capabilities which are available in the Kinesis Streams console. Select S ervices from the AWS console page, type in K inesis a nd select it from the drop down menu.

20/40



Under “K inesis Data Streams ” select “s imFlightData ”.

Select “M onitoring ” from the top tab

2.

The monitoring tab provides a number of useful graphs. Specifically, ensure the following 21/40



● ● ●

Put Record Success (Percent) is consistently at 1.0 (100%) Incoming Data (Count) Sum has approx 4,200 record per minute Write Throughput Exceeded (Count / Request) — Average Always 0

Task 4: Setup the AWS Lambda processing function and connect it to Amazon Kinesis. 1. Following the instructions in Activity 1 Task 2, return the CloudFormation console and Select “C reate Stack ”. 2. Specify the following S3 Template URL. https://s3apsoutheast2.amazonaws.com/reinventabd322workshop/lambdaFunct ionAndDDBStack.json

3.

This CloudFormation template will create an AWS Lambda Function and an Amazon DynamoDB table. The Amazon Kinesis Stream ARN will be required. The other parameters must be left as provided. The stack name must be “s imDataLambdaAndDynamoDB ”.

22/40



4. 5.

Select “N ext ”. The Options page can remain unchanged. Select “N ext ”. As we are creating a number of IAM roles and policies during this step, click “I acknowledge that AWS CloudFormation might create IAM resources. ”

6.

Next, select “C reate ” to start the creation of the CloudFormation Stack. You will be returned to the CloudFormation Console where the status of the stack creation can be monitored.

7.

Click the “s imDataLambdaAndDynamoDB ” row, followed by selecting the “R esources ” Tab.

23/40



In the “P rocessKinesisFunction ” row, identify the P hysical ID column that contains the Lambda function just created. This will take us to the Lambda console. Task 5: Create a trigger for the AWS Lambda function 1. The Lambda console allows configuration of the function itself, event sources as triggers and monitoring of the innovation of the Lambda function. Select “T riggers ”.

2.

3.

As no trigger has been created, there will be no trigger yet. The simDataLambdaAndDynamoDBProcessKinesisFunctionXXXX function also writes data into a DynamoDB table. This table will be used later as a data source for the User Interface, via an API gateway. As there is no trigger configured it is expected that there would be no data in the DynamoDB table. Select S ervices from the AWS console page, type in D ynamo a nd select it from the drop down menu.

Select “T ables ” from the DynamoDB Console to show the list of DynamoDB tables available.

24/40



Select “s imDataStore ”

Followed by selecting “I tems ” from the tab menu.

Observe that there is no data in the DynamoDB table.

4.

Following previous instructions, return to the Lambda Console and select the “ simDataLambdaAndDynamoDBProcessKinesisFunction”. 25/40



Select “T riggers ”.

Select “A dd Trigger ”.

This will display a popup window, in which you can select the AWS service that is the trigger for this Lambda function. Selecting Kinesis will attach a Kinesis Stream to this Lambda function, triggering it as per the configuration supplied.

Clicking in the outlined box, will display a drop down, from which Kinesis can be selected.

26/40



Once, selected, configure the Kinesis trigger. The Kinesis Stream can be selected from the Kinesis stream drop down. Select s imFlightData. Batch Size refers to the number of Kinesis records required before the Lambda function will trigger. As there is a requirement to process each simulator data element, a batch size of 1 should be used. Starting Position allows for the configuration of where the trigger should start start iterating through the Kinesis shard. There are a number of options (refer h ere for details). Select L atest . This will start reading just after the most recent record in the shard, so that the most recent data in the shard is being read.

27/40



Select “S ubmit ” to create the trigger. The trigger has been created. This can be checked by ensuring the following

It can often take a minute or two for the trigger to be instantiated and operational. Refresh the triggers by select “R efresh triggers ”.

After a few moments, the triggers will update to show that the Lambda function is processing the Kinesis data.

Task 5: Validate the trigger is functional 1. With the trigger created it is possible to validate that the Lambda function is processing the Kinesis stream correctly. Select “M onitoring ” 28/40



Review all the CloudWatch metrics, specifically I nvocation count and a nd I nvocation Errors. I t is expected that that Invocation count will be increasing, whilst the invocation errors should remain at 0. Note: It may take a few minutes for these metrics to become available. Please use the refresh icon to refresh the view.

Task 6: View the data in DynamoDB 1 Select S ervices from the AWS console page, type in D ynamo a nd select it from the drop down menu.

29/40



Select “T ables ” from the DynamoDB Console to show the list of DynamoDB tables available.

Select “s imDataStore ”

Followed by selecting “I tems ” from the tab menu.

Flight data will now be available in the DynamoDB table. 30/40



There will be only a single row in the Dynamo DB table. This row will be updated with the latest data that has streamed through the processing pipeline. What has been achieved so far. In activity 2 the workstation was used to consume the stream data from the simulator. This data was injected into Kinesis Streams, where it used as trigger for a Lambda function. This Lambda function writes each “row” of the streaming data into a DynamoDB table for later consumption. END OF ACTIVITY 2

31/40



Activity 3: Exposing flight data via an API Allocated Time: 15 minutes Introduction With data being generated and streamed via the workstation into Kinesis, triggering a lambda function and updating a DynamoDB table, exposing this data to via an API Gateway RESTful interface is required. Approach An AWS API Gateway will be created as an endpoint for the RESTful interface required to deliver the data to a client. This data is sourced from DynamoDB via a Lambda function that is triggered by a GET call to the API Gateway. Prerequisites 1. Activity 2 and all tasks must be completed prior to commencing this activity.

Task 1: Create the API Gateway using CloudFormation 1. Using the CloudFormation console, select C reate Stack.

2.

AWS offers the option to create a stack via a template designer, from a preexisting sample template, or to supply your own via an S3 URL. Select “ Specify an Amazon S3 template URL” and provide the following URL https://s3apsoutheast2.amazonaws.com/reinventabd322workshop/APIGatewayA ndLambdaStack.json

32/40



3.

Specifying details about your stack will allow you to provide a S tack Name a nd configure other parameters exposed by the template provided. Use “s imDataAPIGateway ” f or the Stack Name .

Select “N ext ”. There is no need to make an changes on the O ptions page. Select “N ext ”. 4.

As we are creating a number of IAM roles and policies during this step, click “I acknowledge that AWS CloudFormation might create IAM resources. ”

33/40



5.

Next, select “C reate ” to start the creation of the CloudFormation Stack. After some time, the status of your CloudFormation stack will change from “C REATE_IN_PROGRESS ” to “C REATE_COMPLETE ”. You may need to use the refresh button update the status. This action may time a few minutes to complete. When the status has changed to “C REATE_COMPLETE ” access the O utputs t ab and note the APIUrl. This will be required in Activity 5.

Task 2: Validating the RESTful interface and the data payload 1. Using the workstation SSH session, validate that the API Gateway is returning the expected data.

This should return a JSON object that represents the latest flight data.

2.

The workstation includes a tools called jq. This tool (more information h ere ) is like sed for JSON data you can use it to slice and filter and map and transform structured data with 34/40



the same ease that sed, awk, grep and friends let you play with text. It also makes JSON output more readable.

This will return a nicely formatted JSON string.

35/40



What has been achieved so far.

CloudFormation was used to instantiate an API gateway service, which is backed by Lambda. END OF ACTIVITY 3

36/40



Activity 4: Configure and view the User Interface Allocated Time: 15 minutes Introduction With data being generated and streamed via the workstation into Kinesis, triggering a lambda function and updating a DynamoDB table, exposing this data to via an API Gateway RESTful interface is required. Approach An AWS API Gateway will be created as an endpoint for the RESTful interface required to deliver the data to a client. This data is sourced from DynamoDB via a Lambda function that is triggered by a GET call to the API Gateway. Prerequisites 1. Activity 3 and all tasks must be completed prior to commencing this activity. Task 1: Configure the UI with the API Gateway 1. In activity 3 an API gateway was deployed along with an accompanying Lambda function. Once deployed, the URL (APIUrl) was noted. This URL is now required. 2. Returning to the SSH session established to the simWorkstation the API gateway URL will be configured. 3. Change directories into /simDataUI. This is the location of the User Interface. 4.

app.js location located in the js/ directory will need to be modified to have the API Gateway URL configured. This will be done using the nano text editor.

5.

app.js is the main javascript application of the User Interface. It is responsible for orchestrating the fetching of data from the API gateway and updating the visual aspects of the User Interface. 37/40



6.

On line 2 (see above) replace the t ext with the APIUrl noted earlier.

7.

The Ctrl+X key combination will exit nano, first asking to save the changes. Enter “C trlX ”.

8.

When prompted to “S ave modified buffer ”, enter “Y ” .

9.

Hit Return/Enter to save the file

38/40



Task 2: Starting a web server for the User Interface

1.

2.

To serve the User Interface, Python SimpleHTTPServer (more information can be found here ) will be used. At the command prompt enter “p ython m SimpleHTTPServer 80 ”.

This will start a small webserver on port 80 capable of serving content of the simDataUI directory Task 3: View the User Interface 1. Using the IP address of the simWorkstation browse to http://.

2.

The User Interface should be visible.

39/40



What has been achieved so far.

Your browser is connected to the the user interface on “simulatorWorkstation” which is r eceiving current telemetry data from API Gateway you set up in Activity3. This data is being fed by the process running on “simDataProducer” into the kinesis stream which is using Lambda to write into the DynamoDB table you set up in Activity 2. END OF ACTIVITY 4

40/40