Mar 14, 2017 - Google Cloud Platform. 7 http://bit.ly/tf-strata bit.ly/tensorflow-workshop data algorithm insight. What
Getting Started With TensorFlow Strata
March 14, 2017
Amy Unruh and Yufeng Guo
Your guides These slides: http://bit.ly/tf-strata
Amy
[email protected] @amygdala http://bit.ly/tf-strata
Yufeng
[email protected] @YufengG bit.ly/tensorflow-workshop
Welcome and Logistics
Google Cloud Platform
3
Overview of the Tutorial ● ● ● ●
Intro and Demos Getting started with TensorFlow coding: basic concepts Linear Classification in TensorFlow (MNIST) Building a DNN Classifier using higher-level libs: a more accurate MNIST model
● “Wide & Deep” learning with TensorFlow: census data ● Wrapup http://bit.ly/tf-strata
bit.ly/tensorflow-workshop Google Cloud Platform
4
Slides: http://bit.ly/tf-strata GitHub: https://github.com/amygdala/tensorflow-workshop Optional docker image for style transfer demos: https://github.com/random-forests/WTM http://bit.ly/tf-strata
bit.ly/tensorflow-workshop
Brief intro to some NN concepts
Google Cloud Platform
6
What is Machine Learning?
data
algorithm
http://bit.ly/tf-strata
insight
bit.ly/tensorflow-workshop Google Cloud Platform
7
What is Machine Learning? “Field of study that gives computers the ability to learn without being explicitly programmed".
data
algorithm
http://bit.ly/tf-strata
insight
bit.ly/tensorflow-workshop Google Cloud Platform
8
What is Machine Learning? “You can think of ML as programming with data instead of instructions. The system learns from the data so it can react correctly to new data.".
data
algorithm
http://bit.ly/tf-strata
insight
bit.ly/tensorflow-workshop Google Cloud Platform
9
What is Machine Learning? But: http://research.google.com/pubs/pub43146.html ("Machine Learning: The High Interest Credit Card of Technical Debt")
data
algorithm
http://bit.ly/tf-strata
insight
bit.ly/tensorflow-workshop Google Cloud Platform
10
Deep Learning Current state of the art in many tasks: Image: classification, captioning, colorization Text: translation, parsing, summarization Speech: recognition, generation Games: AlphaGo, Atari, Mario Kart Many more.
Why now? ● Performance improvements: GPUs & Cloud computing ● Algorithmic improvements ● Larger datasets ● Excellent tools
Open Source Models github.com/tensorflow/models
Show and Tell
https://research.googleblog.com/2016/09/show-and-tell-image-captioning-open.html
Parsey McParseface
我
订了
一张
票
去
Google
https://research.googleblog.com/2016/05/announcing-syntaxnet-worlds-most.html
Parsey’s Cousins, in 40 languages
https://research.googleblog.com/2016/08/meet-parseys-cousins-syntax-for-40.html
Translation
https://research.googleblog.com/2016/09/a-neural-network-for-machine.html
Summarization Original text ● Alice and Bob took the train to visit the zoo. They saw a baby giraffe, a lion, and a flock of colorful tropical birds. Abstractive summary ● Alice and Bob visited the zoo and saw animals and birds.
https://research.googleblog.com/2016/08/text-summarization-with-tensorflow.html
Inception: Image classification
An Alaskan Malamute (left) and a Siberian Husky (right). Images from Wikipedia.
https://research.googleblog.com/2016/08/improving-inception-and-image.html
Transfer Learning
Most of the network is unchanged Only the last layer of weights and the outputs are updated Image: neuralnetworksanddeeplearning.com
Bootstrapping with the Inception model
22
Detection of Diabetic Eye Disease
https://research.googleblog.com/2016/11/deep-learning-for-detection-of-diabetic.html
Skin Cancer Image Classification Benign
Malignant http://www.nature.com/nature/journal/v542/n7639/full/nature21056.html
Hug? Don’t Hug? https://goo.gl/zB8TrG
25
TensorFlow for Poets
Codelab goo.gl/xGsB9d Video goo.gl/KewA03
Shared Research in TensorFlow Inception https://research.googleblog.com/2016/08/improving-inception-and-image.html Show and Tell https://research.googleblog.com/2016/09/show-and-tell-image-captioning-open.html Parsey McParseface https://research.googleblog.com/2016/05/announcing-syntaxnet-worlds-most.html Translation https://research.googleblog.com/2016/09/a-neural-network-for-machine.html Summarization https://research.googleblog.com/2016/08/text-summarization-with-tensorflow.html Pathology https://research.googleblog.com/2017/03/assisting-pathologists-in-detecting.html
Many more: https://github.com/tensorflow/models/
Style Transfer
goo.gl/WPJtVe
Image source - Wikipedia
Fast Style Transfer
From https://github.com/lengstrom/fast-style-transfer/
From https://github.com/lengstrom/fast-style-transfer/
From: A Matlab Plugin to Visualize Neurons from Deep Models, Donglai Wei et. al.
https://github.com/lengstrom/fast-style-transfer/ https://research.googleblog.com/2016/10/supercharging-style-transfer.html
Multistyle Transfer with Project Magenta
https://en.wikipedia.org/
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
Drive folder goo.gl/1UDjBE 30
31
Blending styles
Style 0
Style 31
Blended
Project Magenta
ht
Can ML generate compelling media?
A
● ●
Music Images and Video
X
●
Text (Jokes, Stories)
tt
●
Interestingness/Surprise
●
Structure learning
●
Measure success
magenta.tensorflow.org
Check out this great talk
goo.gl/ZFzpzu
Learn more Google Research Blog goo.gl/T3iD6q Paper goo.gl/6fS16m
What’s TensorFlow? (and why is it so great for ML?)
Google Cloud Platform
46
A multidimensional array.
A graph of operations.
A multidimensional array.
A graph of operations.
Operates over tensors: n-dimensional arrays Using a flow graph: data flow computation framework ●
Flexible, intuitive construction
●
automatic differentiation
●
Support for threads, queues, and asynchronous computation; distributed runtime
●
Train on CPUs, GPUs, ...and coming soon, TPUS...
●
Run wherever you like
https://cloudplatform.googleblog.com/2016 /05/Google-supercharges-machine-learnin g-tasks-with-custom-chip.html 48
Tensors - generalized matrices Tensors have a Shape that’s described with a vector.
[ 10000, 256, 256, 3 ] ● ● ● ●
http://bit.ly/tf-strata
10000 Images Each Image has 256 Rows Each Row has 256 Pixels Each Pixel has 3 channels (RGB)
Google Cloud Platform
49
Computation is a dataflow graph
Graph of Nodes, also called Operations or ops.
biases
Add
weights MatMul examples
labels
Relu Xent
Computation is a dataflow graph
Edges are N-dimensional arrays: Tensors
biases
Add
weights MatMul examples
labels
with
s r o s ten
Relu Xent
Computation is a dataflow graph
'Biases' is a variable
e t a t ith s
w
Some ops compute gradients
−= updates biases
biases
...
learning rate
Add
...
Mul
−=
Build a graph; then run it.
From: http://googleresearch.blogspot.com/2016_03_01_archive.html
Google Cloud Platform
54
Python Frontend
C++ Frontend
...
TensorFlow Distributed Execution Engine CPU
GPU
Android
iOS
...
Build models
Layers Python Frontend
C++ Frontend
...
TensorFlow Distributed Execution Engine CPU
GPU
Android
iOS
...
Train and evaluate models
Keras Model
Estimator
Build models
Layers Python Frontend
C++ Frontend
...
TensorFlow Distributed Execution Engine CPU
GPU
Android
iOS
...
Models in a box
Canned Estimators
Train and evaluate models
Keras Model
Estimator
Build models
Layers Python Frontend
C++ Frontend
...
TensorFlow Distributed Execution Engine CPU
GPU
Android
iOS
...
Tensorboard: Graph Visualization
http://bit.ly/tf-strata
Google Cloud Platform
59
TensorBoard
https://www.tensorflow.org/versions/r0.12/how_tos/embedding_viz/index.html
playground.tensorflow.org
TensorFlow API Documentation: https://www.tensorflow.org/api_docs/
Google Cloud Platform
62
A first look at some code: Creating and running a TensorFlow graph
Google Cloud Platform
63
Define the TensorFlow graph
Google Cloud Platform
64
Create a TensorFlow graph Follow along at: https://github.com/amygdala/tensorflow-workshop/tree/master/workshop_sections/starter_tf_graph
import numpy as np import tensorflow as tf # a constant m1 = np.array([[1.,2.], [3.,4.], [5.,6.], [7., 8.]], dtype=np.float32) # Input data placeholder m1_input = tf.placeholder(tf.int32, shape=[4,2])
http://bit.ly/tf-strata
Google Cloud Platform
65
Create a TensorFlow graph, cont. Follow along at: https://github.com/amygdala/tensorflow-workshop/tree/master/workshop_sections/starter_tf_graph
m2 = tf.Variable(tf.random_uniform([2,3], -1.0, 1.0)) m3 = tf.matmul(m1_input, m2)
# Add variable initializer op. init = tf.initialize_all_variables()
http://bit.ly/tf-strata
Google Cloud Platform
66
Running the graph (deferred execution)
Google Cloud Platform
67
Running a TensorFlow graph in a session Follow along at: https://github.com/amygdala/tensorflow-workshop/tree/master/workshop_sections/starter_tf_graph
with tf.Session() as session: # We must initialize all variables before we use them. init.run() feed_dict = {m1_input: m1} result = session.run([m3], feed_dict=feed_dict) print("\nresult: {}\n".format(result))
http://bit.ly/tf-strata
Google Cloud Platform
68
We just used core TensorFlow concepts...
Google Cloud Platform
69
Core TensorFlow data structures and concepts... - Graph: A TensorFlow computation, represented as a dataflow graph. - collection of ops that may be executed together as a group - Operation: a graph node that performs computation on tensors - Tensor: a handle to one of the outputs of an Operation - provides a means of computing the value in a TensorFlow Session. bit.ly/tensorflow-workshop
http://bit.ly/tf-strata Google Cloud Platform
70
Core TensorFlow data structures and concepts - Constants - Placeholders: must be fed with data on execution - Variables: a modifiable tensor that lives in TensorFlow's graph of interacting operations. - Session: encapsulates the environment in which Operation objects are executed, and Tensor objects are evaluated.
bit.ly/tensorflow-workshop
http://bit.ly/tf-strata Google Cloud Platform
71
Let’s add more nodes to this graph.. Follow along at: https://github.com/amygdala/tensorflow-workshop/tree/master/workshop_sections/starter_tf_graph
● Add m3 to itself ● Store the result in m4 ● Return the results for both m3 and m4 Useful link: https://www.tensorflow.org/api_guides/python/math_ops
http://bit.ly/tf-strata
bit.ly/tensorflow-workshop Google Cloud Platform
72
MNIST: The “Hello World” Of ML
Google Cloud Platform
73
Computer Vision -- MNIST
http://bit.ly/tf-strata
bit.ly/tensorflow-workshop Google Cloud Platform
74
What we see
What the computer “sees”
Training data
training_data training_labels testing_data testing_labels
Testing data
= = = =
mnist.train.images mnist.train.labels mnist.test.images mnist.test.labels
A simple version of MNIST
Google Cloud Platform
77
Linear Regression
f(x)= mx + b
Linear Classification: Apply a Logistic Function f(x)= mx + b * L
f(x) = predicted value y = 0 if f(x) < 0 y = f(x) otherwise
http://bit.ly/tf-strata
bit.ly/tensorflow-workshop Google Cloud Platform
80
http://bit.ly/tf-strata
bit.ly/tensorflow-workshop Google Cloud Platform
81
Minimize loss: optimizers error
tf.train.GradientDescentOptimizer
function minimum
parameters (weights, biases)
Training = computing values for the variables W and b
http://bit.ly/tf-strata
bit.ly/tensorflow-workshop Google Cloud Platform
83
Because TensorFlow knows the entire graph of your computations, it can automatically use the backpropagation algorithm to efficiently determine how your variables affect the cost (loss function) you ask it to minimize. Then it can apply your choice of optimization algorithm to modify the variables and reduce the cost. http://bit.ly/tf-strata
bit.ly/tensorflow-workshop Google Cloud Platform
84
“one-hot” decoding
http://bit.ly/tf-strata
bit.ly/tensorflow-workshop Google Cloud Platform
85
Common TF NN pattern: - Create the model (inference) graph - Define the loss function - specify the optimizer and learning rate → training step op - In a training loop, call sess.run([train_step,..], feed_dict={....}) where feed_dict maps inputs to placeholder values
bit.ly/tensorflow-workshop
http://bit.ly/tf-strata Google Cloud Platform
86
Lab: A simple version of MNIST Workshop section: mnist_series/ 01_README_mnist_simple.md
Google Cloud Platform
87
Let’s make our MNIST model more accurate... we’ll add some hidden layers (and take a look at TensorBoard)
Google Cloud Platform
88
...
Hidden layers
{ ...
0
1
2
9
One “fully connected” NN layer typically looks like...
Implementation as Matrix Multiplication
ReLu (Rectified Linear unit)
http://bit.ly/tf-strata
bit.ly/tensorflow-workshop Google Cloud Platform
92
Models in a box
Canned Estimators
Train and evaluate models
Keras Model
Estimator
Build models
Layers Python Frontend
C++ Frontend
...
TensorFlow Distributed Execution Engine CPU
GPU
Android
iOS
...
Lab: hidden-layers version of MNIST with TensorFlow’s Estimator APIs Workshop section: mnist_series/mnist_tflearn.md
Google Cloud Platform
94
Tensorboard: Graph Visualization
http://bit.ly/tf-strata
Google Cloud Platform
95
Comparing two different optimizers for DNNClassifier Google Cloud Platform
96
“Wide & Deep”: Using the TensorFlow High-level APIs
Google Cloud Platform
97
Models in a box
Canned Estimators
Train and evaluate models
Keras Model
Estimator
Build models
Layers Python Frontend
C++ Frontend
...
TensorFlow Distributed Execution Engine CPU
GPU
Android
iOS
...
Many levels of abstraction Choose the right one for you: ● Layers, losses, metrics ● Training/Eval loop functions ● Estimator (BaseEstimator) ○
Any model you want, but must separate input from the rest of the model
● Predefined estimators ○ ○
,… Limited configuration options: feature columns, metrics.
http://bit.ly/tf-strata
Google Cloud Platform
99
Typical structure ● Load data ● Set up feature columns ● Create your model ● Run the training loop (fit the model) ● Evaluate your model's accuracy (and other metrics) ● (optional) Predict new examples http://bit.ly/tf-strata
Google Cloud Platform
100
tf.learn high-level structure
http://bit.ly/tf-strata
Google Cloud Platform
101
Motivation - a "magical" food app
http://bit.ly/tf-strata
Google Cloud Platform
102
Launch and Iterate!
● Naive character matching ● Say "Fried chicken" ● Get "Chicken Fried Rice" ● Oops. Now what? ● Machine learning to the rescue! http://bit.ly/tf-strata
Google Cloud Platform
103
v2.0: memorize all the things
● Train a linear TF model
● Your app is gaining traction! http://bit.ly/tf-strata
Google Cloud Platform
104
Problem: Your users are bored!
● Too many
& waffles
● Show me similar, but different food
● Your users are picky
http://bit.ly/tf-strata
Google Cloud Platform
105
v3.0: More generalized recommendations for all
http://bit.ly/tf-strata
Google Cloud Platform
106
No good deed goes unpunished
● Some recommendations are "too general" ○ Irrelevant dishes are being sent ● Your users are still picky
http://bit.ly/tf-strata
Google Cloud Platform
107
No good deed goes unpunished ● 2 types of requests: specific and general ● "iced decaf latte with nonfat milk" != "hot latte with whole milk" ● “seafood” or “italian food” or “fast food” ● How to balance this?
http://bit.ly/tf-strata
Google Cloud Platform
108
v4.0: Why not both?
http://bit.ly/tf-strata
Google Cloud Platform
109
Lab: Wide and Deep: Using TensorFlow’s high-level APIs Workshop section: wide_n_deep
Google Cloud Platform
110
Meet our dataset: US Census Data ● Just as exciting as chicken and waffles ● Task: predict the probability that the individual has an annual income of over 50,000 dollars ● Over 32k training examples ● Was extracted from the 1994 US Census by Barry Becker. http://bit.ly/tf-strata
Google Cloud Platform
111
Meet our dataset: US Census Data Column Name
Type
Description
age
Continuous
The age of the individual
workclass
Categorical
The type of employer the individual has (government, military, private, etc.).
fnlwgt
Continuous
The number of people the census takers believe that observation represents (sample weight). This variable will not be used.
education
Categorical
The highest level of education achieved for that individual.
education_num
Continuous
The highest level of education in numerical form.
marital_status
Categorical
Marital status of the individual.
http://bit.ly/tf-strata
Google Cloud Platform
112
Meet our dataset: US Census Data Column Name
Type
Description
occupation
Categorical
The occupation of the individual.
relationship
Categorical
Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.
race
Categorical
White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black.
gender
Categorical
Female, Male.
capital_gain
Continuous
Capital gains recorded.
capital_loss
Continuous
Capital Losses recorded.
http://bit.ly/tf-strata
Google Cloud Platform
113
Meet our dataset: US Census Data Column Name
Type
Description
hours_per_week
Continuous
Hours worked per week.
native_country
Categorical
Country of origin of the individual.
income_bracket
Categorical
">50K" or "