Getting Started With TensorFlow

5 downloads 441 Views 12MB Size Report
Mar 14, 2017 - Google Cloud Platform. 7 http://bit.ly/tf-strata bit.ly/tensorflow-workshop data algorithm insight. What
Getting Started With TensorFlow Strata

March 14, 2017

Amy Unruh and Yufeng Guo

Your guides These slides: http://bit.ly/tf-strata

Amy [email protected] @amygdala http://bit.ly/tf-strata

Yufeng [email protected] @YufengG bit.ly/tensorflow-workshop

Welcome and Logistics

Google Cloud Platform

3

Overview of the Tutorial ● ● ● ●

Intro and Demos Getting started with TensorFlow coding: basic concepts Linear Classification in TensorFlow (MNIST) Building a DNN Classifier using higher-level libs: a more accurate MNIST model

● “Wide & Deep” learning with TensorFlow: census data ● Wrapup http://bit.ly/tf-strata

bit.ly/tensorflow-workshop Google Cloud Platform

4

Slides: http://bit.ly/tf-strata GitHub: https://github.com/amygdala/tensorflow-workshop Optional docker image for style transfer demos: https://github.com/random-forests/WTM http://bit.ly/tf-strata

bit.ly/tensorflow-workshop

Brief intro to some NN concepts

Google Cloud Platform

6

What is Machine Learning?

data

algorithm

http://bit.ly/tf-strata

insight

bit.ly/tensorflow-workshop Google Cloud Platform

7

What is Machine Learning? “Field of study that gives computers the ability to learn without being explicitly programmed".

data

algorithm

http://bit.ly/tf-strata

insight

bit.ly/tensorflow-workshop Google Cloud Platform

8

What is Machine Learning? “You can think of ML as programming with data instead of instructions. The system learns from the data so it can react correctly to new data.".

data

algorithm

http://bit.ly/tf-strata

insight

bit.ly/tensorflow-workshop Google Cloud Platform

9

What is Machine Learning? But: http://research.google.com/pubs/pub43146.html ("Machine Learning: The High Interest Credit Card of Technical Debt")

data

algorithm

http://bit.ly/tf-strata

insight

bit.ly/tensorflow-workshop Google Cloud Platform

10

Deep Learning Current state of the art in many tasks: Image: classification, captioning, colorization Text: translation, parsing, summarization Speech: recognition, generation Games: AlphaGo, Atari, Mario Kart Many more.

Why now? ● Performance improvements: GPUs & Cloud computing ● Algorithmic improvements ● Larger datasets ● Excellent tools

Open Source Models github.com/tensorflow/models

Show and Tell

https://research.googleblog.com/2016/09/show-and-tell-image-captioning-open.html

Parsey McParseface



订了

一张





Google

https://research.googleblog.com/2016/05/announcing-syntaxnet-worlds-most.html

Parsey’s Cousins, in 40 languages

https://research.googleblog.com/2016/08/meet-parseys-cousins-syntax-for-40.html

Translation

https://research.googleblog.com/2016/09/a-neural-network-for-machine.html

Summarization Original text ● Alice and Bob took the train to visit the zoo. They saw a baby giraffe, a lion, and a flock of colorful tropical birds. Abstractive summary ● Alice and Bob visited the zoo and saw animals and birds.

https://research.googleblog.com/2016/08/text-summarization-with-tensorflow.html

Inception: Image classification

An Alaskan Malamute (left) and a Siberian Husky (right). Images from Wikipedia.

https://research.googleblog.com/2016/08/improving-inception-and-image.html

Transfer Learning

Most of the network is unchanged Only the last layer of weights and the outputs are updated Image: neuralnetworksanddeeplearning.com

Bootstrapping with the Inception model

22

Detection of Diabetic Eye Disease

https://research.googleblog.com/2016/11/deep-learning-for-detection-of-diabetic.html

Skin Cancer Image Classification Benign

Malignant http://www.nature.com/nature/journal/v542/n7639/full/nature21056.html

Hug? Don’t Hug? https://goo.gl/zB8TrG

25

TensorFlow for Poets

Codelab goo.gl/xGsB9d Video goo.gl/KewA03

Shared Research in TensorFlow Inception https://research.googleblog.com/2016/08/improving-inception-and-image.html Show and Tell https://research.googleblog.com/2016/09/show-and-tell-image-captioning-open.html Parsey McParseface https://research.googleblog.com/2016/05/announcing-syntaxnet-worlds-most.html Translation https://research.googleblog.com/2016/09/a-neural-network-for-machine.html Summarization https://research.googleblog.com/2016/08/text-summarization-with-tensorflow.html Pathology https://research.googleblog.com/2017/03/assisting-pathologists-in-detecting.html

Many more: https://github.com/tensorflow/models/

Style Transfer

goo.gl/WPJtVe

Image source - Wikipedia

Fast Style Transfer

From https://github.com/lengstrom/fast-style-transfer/

From https://github.com/lengstrom/fast-style-transfer/

From: A Matlab Plugin to Visualize Neurons from Deep Models, Donglai Wei et. al.

https://github.com/lengstrom/fast-style-transfer/ https://research.googleblog.com/2016/10/supercharging-style-transfer.html

Multistyle Transfer with Project Magenta

https://en.wikipedia.org/

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

Drive folder goo.gl/1UDjBE 30

31

Blending styles

Style 0

Style 31

Blended

Project Magenta

ht

Can ML generate compelling media?

A

● ●

Music Images and Video

X



Text (Jokes, Stories)

tt



Interestingness/Surprise



Structure learning



Measure success

magenta.tensorflow.org

Check out this great talk

goo.gl/ZFzpzu

Learn more Google Research Blog goo.gl/T3iD6q Paper goo.gl/6fS16m

What’s TensorFlow? (and why is it so great for ML?)

Google Cloud Platform

46

A multidimensional array.

A graph of operations.

A multidimensional array.

A graph of operations.

Operates over tensors: n-dimensional arrays Using a flow graph: data flow computation framework ●

Flexible, intuitive construction



automatic differentiation



Support for threads, queues, and asynchronous computation; distributed runtime



Train on CPUs, GPUs, ...and coming soon, TPUS...



Run wherever you like

https://cloudplatform.googleblog.com/2016 /05/Google-supercharges-machine-learnin g-tasks-with-custom-chip.html 48

Tensors - generalized matrices Tensors have a Shape that’s described with a vector.

[ 10000, 256, 256, 3 ] ● ● ● ●

http://bit.ly/tf-strata

10000 Images Each Image has 256 Rows Each Row has 256 Pixels Each Pixel has 3 channels (RGB)

Google Cloud Platform

49

Computation is a dataflow graph

Graph of Nodes, also called Operations or ops.

biases

Add

weights MatMul examples

labels

Relu Xent

Computation is a dataflow graph

Edges are N-dimensional arrays: Tensors

biases

Add

weights MatMul examples

labels

with

s r o s ten

Relu Xent

Computation is a dataflow graph

'Biases' is a variable

e t a t ith s

w

Some ops compute gradients

−= updates biases

biases

...

learning rate

Add

...

Mul

−=

Build a graph; then run it.

From: http://googleresearch.blogspot.com/2016_03_01_archive.html

Google Cloud Platform

54

Python Frontend

C++ Frontend

...

TensorFlow Distributed Execution Engine CPU

GPU

Android

iOS

...

Build models

Layers Python Frontend

C++ Frontend

...

TensorFlow Distributed Execution Engine CPU

GPU

Android

iOS

...

Train and evaluate models

Keras Model

Estimator

Build models

Layers Python Frontend

C++ Frontend

...

TensorFlow Distributed Execution Engine CPU

GPU

Android

iOS

...

Models in a box

Canned Estimators

Train and evaluate models

Keras Model

Estimator

Build models

Layers Python Frontend

C++ Frontend

...

TensorFlow Distributed Execution Engine CPU

GPU

Android

iOS

...

Tensorboard: Graph Visualization

http://bit.ly/tf-strata

Google Cloud Platform

59

TensorBoard

https://www.tensorflow.org/versions/r0.12/how_tos/embedding_viz/index.html

playground.tensorflow.org

TensorFlow API Documentation: https://www.tensorflow.org/api_docs/

Google Cloud Platform

62

A first look at some code: Creating and running a TensorFlow graph

Google Cloud Platform

63

Define the TensorFlow graph

Google Cloud Platform

64

Create a TensorFlow graph Follow along at: https://github.com/amygdala/tensorflow-workshop/tree/master/workshop_sections/starter_tf_graph

import numpy as np import tensorflow as tf # a constant m1 = np.array([[1.,2.], [3.,4.], [5.,6.], [7., 8.]], dtype=np.float32) # Input data placeholder m1_input = tf.placeholder(tf.int32, shape=[4,2])

http://bit.ly/tf-strata

Google Cloud Platform

65

Create a TensorFlow graph, cont. Follow along at: https://github.com/amygdala/tensorflow-workshop/tree/master/workshop_sections/starter_tf_graph

m2 = tf.Variable(tf.random_uniform([2,3], -1.0, 1.0)) m3 = tf.matmul(m1_input, m2)

# Add variable initializer op. init = tf.initialize_all_variables()

http://bit.ly/tf-strata

Google Cloud Platform

66

Running the graph (deferred execution)

Google Cloud Platform

67

Running a TensorFlow graph in a session Follow along at: https://github.com/amygdala/tensorflow-workshop/tree/master/workshop_sections/starter_tf_graph

with tf.Session() as session: # We must initialize all variables before we use them. init.run() feed_dict = {m1_input: m1} result = session.run([m3], feed_dict=feed_dict) print("\nresult: {}\n".format(result))

http://bit.ly/tf-strata

Google Cloud Platform

68

We just used core TensorFlow concepts...

Google Cloud Platform

69

Core TensorFlow data structures and concepts... - Graph: A TensorFlow computation, represented as a dataflow graph. - collection of ops that may be executed together as a group - Operation: a graph node that performs computation on tensors - Tensor: a handle to one of the outputs of an Operation - provides a means of computing the value in a TensorFlow Session. bit.ly/tensorflow-workshop

http://bit.ly/tf-strata Google Cloud Platform

70

Core TensorFlow data structures and concepts - Constants - Placeholders: must be fed with data on execution - Variables: a modifiable tensor that lives in TensorFlow's graph of interacting operations. - Session: encapsulates the environment in which Operation objects are executed, and Tensor objects are evaluated.

bit.ly/tensorflow-workshop

http://bit.ly/tf-strata Google Cloud Platform

71

Let’s add more nodes to this graph.. Follow along at: https://github.com/amygdala/tensorflow-workshop/tree/master/workshop_sections/starter_tf_graph

● Add m3 to itself ● Store the result in m4 ● Return the results for both m3 and m4 Useful link: https://www.tensorflow.org/api_guides/python/math_ops

http://bit.ly/tf-strata

bit.ly/tensorflow-workshop Google Cloud Platform

72

MNIST: The “Hello World” Of ML

Google Cloud Platform

73

Computer Vision -- MNIST

http://bit.ly/tf-strata

bit.ly/tensorflow-workshop Google Cloud Platform

74

What we see

What the computer “sees”

Training data

training_data training_labels testing_data testing_labels

Testing data

= = = =

mnist.train.images mnist.train.labels mnist.test.images mnist.test.labels

A simple version of MNIST

Google Cloud Platform

77

Linear Regression

f(x)= mx + b

Linear Classification: Apply a Logistic Function f(x)= mx + b * L

f(x) = predicted value y = 0 if f(x) < 0 y = f(x) otherwise

http://bit.ly/tf-strata

bit.ly/tensorflow-workshop Google Cloud Platform

80

http://bit.ly/tf-strata

bit.ly/tensorflow-workshop Google Cloud Platform

81

Minimize loss: optimizers error

tf.train.GradientDescentOptimizer

function minimum

parameters (weights, biases)

Training = computing values for the variables W and b

http://bit.ly/tf-strata

bit.ly/tensorflow-workshop Google Cloud Platform

83

Because TensorFlow knows the entire graph of your computations, it can automatically use the backpropagation algorithm to efficiently determine how your variables affect the cost (loss function) you ask it to minimize. Then it can apply your choice of optimization algorithm to modify the variables and reduce the cost. http://bit.ly/tf-strata

bit.ly/tensorflow-workshop Google Cloud Platform

84

“one-hot” decoding

http://bit.ly/tf-strata

bit.ly/tensorflow-workshop Google Cloud Platform

85

Common TF NN pattern: - Create the model (inference) graph - Define the loss function - specify the optimizer and learning rate → training step op - In a training loop, call sess.run([train_step,..], feed_dict={....}) where feed_dict maps inputs to placeholder values

bit.ly/tensorflow-workshop

http://bit.ly/tf-strata Google Cloud Platform

86

Lab: A simple version of MNIST Workshop section: mnist_series/ 01_README_mnist_simple.md

Google Cloud Platform

87

Let’s make our MNIST model more accurate... we’ll add some hidden layers (and take a look at TensorBoard)

Google Cloud Platform

88

...

Hidden layers

{ ...

0

1

2

9

One “fully connected” NN layer typically looks like...

Implementation as Matrix Multiplication

ReLu (Rectified Linear unit)

http://bit.ly/tf-strata

bit.ly/tensorflow-workshop Google Cloud Platform

92

Models in a box

Canned Estimators

Train and evaluate models

Keras Model

Estimator

Build models

Layers Python Frontend

C++ Frontend

...

TensorFlow Distributed Execution Engine CPU

GPU

Android

iOS

...

Lab: hidden-layers version of MNIST with TensorFlow’s Estimator APIs Workshop section: mnist_series/mnist_tflearn.md

Google Cloud Platform

94

Tensorboard: Graph Visualization

http://bit.ly/tf-strata

Google Cloud Platform

95

Comparing two different optimizers for DNNClassifier Google Cloud Platform

96

“Wide & Deep”: Using the TensorFlow High-level APIs

Google Cloud Platform

97

Models in a box

Canned Estimators

Train and evaluate models

Keras Model

Estimator

Build models

Layers Python Frontend

C++ Frontend

...

TensorFlow Distributed Execution Engine CPU

GPU

Android

iOS

...

Many levels of abstraction Choose the right one for you: ● Layers, losses, metrics ● Training/Eval loop functions ● Estimator (BaseEstimator) ○

Any model you want, but must separate input from the rest of the model

● Predefined estimators ○ ○

,… Limited configuration options: feature columns, metrics.

http://bit.ly/tf-strata

Google Cloud Platform

99

Typical structure ● Load data ● Set up feature columns ● Create your model ● Run the training loop (fit the model) ● Evaluate your model's accuracy (and other metrics) ● (optional) Predict new examples http://bit.ly/tf-strata

Google Cloud Platform

100

tf.learn high-level structure

http://bit.ly/tf-strata

Google Cloud Platform

101

Motivation - a "magical" food app

http://bit.ly/tf-strata

Google Cloud Platform

102

Launch and Iterate!

● Naive character matching ● Say "Fried chicken" ● Get "Chicken Fried Rice" ● Oops. Now what? ● Machine learning to the rescue! http://bit.ly/tf-strata

Google Cloud Platform

103

v2.0: memorize all the things

● Train a linear TF model

● Your app is gaining traction! http://bit.ly/tf-strata

Google Cloud Platform

104

Problem: Your users are bored!

● Too many

& waffles

● Show me similar, but different food

● Your users are picky

http://bit.ly/tf-strata

Google Cloud Platform

105

v3.0: More generalized recommendations for all

http://bit.ly/tf-strata

Google Cloud Platform

106

No good deed goes unpunished

● Some recommendations are "too general" ○ Irrelevant dishes are being sent ● Your users are still picky

http://bit.ly/tf-strata

Google Cloud Platform

107

No good deed goes unpunished ● 2 types of requests: specific and general ● "iced decaf latte with nonfat milk" != "hot latte with whole milk" ● “seafood” or “italian food” or “fast food” ● How to balance this?

http://bit.ly/tf-strata

Google Cloud Platform

108

v4.0: Why not both?

http://bit.ly/tf-strata

Google Cloud Platform

109

Lab: Wide and Deep: Using TensorFlow’s high-level APIs Workshop section: wide_n_deep

Google Cloud Platform

110

Meet our dataset: US Census Data ● Just as exciting as chicken and waffles ● Task: predict the probability that the individual has an annual income of over 50,000 dollars ● Over 32k training examples ● Was extracted from the 1994 US Census by Barry Becker. http://bit.ly/tf-strata

Google Cloud Platform

111

Meet our dataset: US Census Data Column Name

Type

Description

age

Continuous

The age of the individual

workclass

Categorical

The type of employer the individual has (government, military, private, etc.).

fnlwgt

Continuous

The number of people the census takers believe that observation represents (sample weight). This variable will not be used.

education

Categorical

The highest level of education achieved for that individual.

education_num

Continuous

The highest level of education in numerical form.

marital_status

Categorical

Marital status of the individual.

http://bit.ly/tf-strata

Google Cloud Platform

112

Meet our dataset: US Census Data Column Name

Type

Description

occupation

Categorical

The occupation of the individual.

relationship

Categorical

Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.

race

Categorical

White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black.

gender

Categorical

Female, Male.

capital_gain

Continuous

Capital gains recorded.

capital_loss

Continuous

Capital Losses recorded.

http://bit.ly/tf-strata

Google Cloud Platform

113

Meet our dataset: US Census Data Column Name

Type

Description

hours_per_week

Continuous

Hours worked per week.

native_country

Categorical

Country of origin of the individual.

income_bracket

Categorical

">50K" or "