Multinomial Classification

view jupyter notebook code

List of Tensorflow 2.0 Tutorials

Following the binary classifier, you can also easily implement a multi-category classifier.


(Review) Logistic Regression

Hypothesis

\[{H(X)} = {1 \over {1+ e^{-XW}}}\]

Cost Function

\[Cost(W) = {1 \over m} {\sum_{i=1}^m c(H(x_{i}), y_{i})}\]

Cross Entropy in Logistic Classification

\[c(H(x), y) = \begin{cases}-log(H(x)) : y=1 \\ -log(1-H(x)) : y=0\end{cases} = -y log(H(x)) - (1-y) log(1-H(x))\]

Minimizing Cost

\[W_{new} = W_{old} - \alpha {\partial \over {\partial W}} Cost(W)\]


Multinomial Classification

Softmax Function (Hypothesis)

\[S(y_{i}) = {e^{y_i} \over \sum_{j=1}^n e^{y_{j}}}\] \[n: Number\ of\ classes \\ i: i\ class\]

Cost Function

\[Cost(W) = {1 \over m} {\sum_{i=1}^m D(S(X_{i}W + b),L_{i})} \\ m: Number\ of\ instances \\ i: i\ instance\]

Cross Entropy in Multinomial Classification

\[D(S, L) = - \sum_{j=1}^n L_{j} log(S(y_{j})) \\ \\ n: Number\ of\ classes \\ \\ j: j\ class\]

Minimizing Cost

\[W_{new} = W_{old} - \alpha {\partial \over {\partial W}} Cost(W)\]


Implement

import tensorflow as tf
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

print("TensorFlow Version: %s" % (tf.__version__))

TensorFlow Version: 2.0.0


Data

x_data = [[1, 2, 1, 1],
          [2, 1, 3, 2],
          [3, 1, 3, 4],
          [4, 1, 5, 5],
          [1, 7, 5, 5],
          [1, 2, 5, 6],
          [1, 6, 6, 6],
          [1, 7, 7, 7]]

y_data = [[0, 0, 1],
          [0, 0, 1],
          [0, 0, 1],
          [0, 1, 0],
          [0, 1, 0],
          [0, 1, 0],
          [1, 0, 0],
          [1, 0, 0]]

x_data = np.array(x_data, dtype=np.float32)
y_data = np.array(y_data, dtype=np.float32)

nb_classes = 3

# Weights
tf.random.set_seed(2020)
W = tf.Variable(tf.random.normal([4, nb_classes], mean=0.0))
b = tf.Variable(tf.random.normal([nb_classes], mean=0.0))

print('# Weights: \n', W.numpy(), '\n\n# Bias: \n', b.numpy())

# Weights:
 [[-0.10099822  0.6847899   1.6258513 ]
 [ 0.88112587 -0.63692456 -0.1427695 ]
 [ 0.82411087 -0.91326994 -0.4510184 ]
 [ 0.58053356  1.3066356  -0.60428965]]

# Bias:
 [ 0.38414612 -0.6159301  -0.5453214 ]

# Learning Rate
learning_rate = 0.01

# Softmax Function
def softmax(X):
    sm = tf.nn.softmax(tf.matmul(x_data, W) + b)
    return sm

# Training
for i in range(10000+1):

    with tf.GradientTape() as tape:

        sm = softmax(x_data)
        cost = tf.reduce_mean(-tf.reduce_sum(y_data*tf.math.log(sm), axis=1))        
        W_grad, b_grad = tape.gradient(cost, [W, b])

        W.assign_sub(learning_rate * W_grad)
        b.assign_sub(learning_rate * b_grad)

    if i % 1000 == 0:
        print(">>> #%s \n Weights: \n%s \n Bias: \n%s \n cost: %s\n" % (i, W.numpy(), b.numpy(), cost.numpy()))

>>> #0
 Weights:
[[-0.11553647  0.6918998   1.6332797 ]
 [ 0.8638605  -0.62458277 -0.13784593]
 [ 0.797113   -0.8949384  -0.44235206]
 [ 0.55236673  1.3261304  -0.5956176 ]]
 Bias:
[ 0.37683368 -0.61232066 -0.5416184 ]
 cost: 5.2257767

>>> #1000
 Weights:
[[-0.8411931   0.75385374  2.2969847 ]
 [-0.07680082 -0.16606328  0.34429583]
 [ 0.618117   -0.6086176  -0.54967654]
 [ 0.24572133  1.326133   -0.28897592]]
 Bias:
[-0.13876517 -0.8163229   0.1779835 ]
 cost: 0.50721896

>>> #2000
 Weights:
[[-1.1914808   0.80435884  2.5967705 ]
 [-0.14087628 -0.18363482  0.42594275]
 [ 0.8594195  -0.52478886 -0.8748073 ]
 [ 0.22371401  1.3248692  -0.26570395]]
 Bias:
[-0.40732294 -0.9690039   0.59922254]
 cost: 0.43953097

>>> #3000
 Weights:
[[-1.464515    0.8519873   2.8221765 ]
 [-0.18783872 -0.19453     0.48380077]
 [ 1.0863445  -0.49375    -1.1327715 ]
 [ 0.16596384  1.3500665  -0.23314708]]
 Bias:
[-0.63186496 -1.08178     0.9365396 ]
 cost: 0.39596182

>>> #4000
 Weights:
[[-1.697922    0.9001363   3.0074346 ]
 [-0.22953543 -0.20061918  0.5315868 ]
 [ 1.3032706  -0.4934833  -1.3499686 ]
 [ 0.09487277  1.3891125  -0.2010959 ]]
 Bias:
[-0.83245873 -1.1711738   1.226527  ]
 cost: 0.36297297

>>> #5000
 Weights:
[[-1.9057775   0.9478083   3.1676147 ]
 [-0.26796636 -0.20388351  0.57328254]
 [ 1.5109538  -0.5109507  -1.5401874 ]
 [ 0.01810047  1.4356134  -0.17081594]]
 Bias:
[-1.0164189 -1.2455534  1.4848666]
 cost: 0.33603936

>>> #6000
 Weights:
[[-2.0948424   0.9941726   3.3103127 ]
 [-0.3038853  -0.20546497  0.6107832 ]
 [ 1.7097456  -0.53860354 -1.7113296 ]
 [-0.06074882  1.485766   -0.1421119 ]]
 Bias:
[-1.1872404 -1.3096143  1.7197511]
 cost: 0.3131696

>>> #7000
 Weights:
[[-2.2689846   1.0387573   3.4398637 ]
 [-0.33769488 -0.20603302  0.645161  ]
 [ 1.8999859  -0.5720022  -1.8681755 ]
 [-0.13977788  1.5373282  -0.11464822]]
 Bias:
[-1.3469772 -1.3662182  1.9360921]
 cost: 0.29330796

>>> #8000
 Weights:
[[-2.4307153   1.0813508   3.5589974 ]
 [-0.3696618  -0.2059924   0.67708796]
 [ 2.0820625  -0.60849404 -2.0137668 ]
 [-0.217931    1.5890023  -0.08818024]]
 Bias:
[-1.4970391 -1.417211   2.1371472]
 cost: 0.2758096

>>> #9000
 Weights:
[[-2.5818226   1.1218913   3.6695557 ]
 [-0.39999598 -0.20558535  0.70701504]
 [ 2.2564096  -0.64648116 -2.1501348 ]
 [-0.29460648  1.6400516  -0.06255731]]
 Bias:
[-1.6384954 -1.4638423  2.325234 ]
 cost: 0.26023802

>>> #10000
 Weights:
[[-2.7236586   1.1604131   3.772864  ]
 [-0.4288627  -0.20496319  0.7352611 ]
 [ 2.4234724  -0.6849797  -2.278707  ]
 [-0.36947724  1.6900569  -0.03769292]]
 Bias:
[-1.7722151 -1.5069835  2.5020947]
 cost: 0.24627577


Predict

predicted = tf.argmax(softmax(x_data), axis=1)
real = tf.argmax(y_data, axis=1)

def acc(predicted, real):
    accuracy = tf.reduce_mean(tf.cast(tf.equal(predicted, real), dtype=tf.float32))
    return accuracy

accuracy = acc(predicted, real).numpy()
print("Accuracy: %s" % accuracy)

Accuracy: 1.0

predicted.numpy()

array([2, 2, 2, 1, 1, 1, 0, 0])

real.numpy()

array([2, 2, 2, 1, 1, 1, 0, 0])


List of Tensorflow 2.0 Tutorials

댓글남기기

참고