ML visual learning
I am learning some basic ML algorithms recently. I prefer to write Python implementations compared to boring formulas, and then visualize them through Matplotlib, a powerful library. For example, the simplest perceptron gradient descent algorithm, given a set of data, can be easily drawn with different colors and shapes:
The figure above depicts the data set[[(3, 3), 1], [(4, 3), 1], [(1, 1), -1]]。
Similarly, for each updated superplane, you can also draw:
The figure above depicts the following separation hyperplane:
w b [3, 3] 1 [2, 2] 0 [1, 1] -1 [0, 0] -2 [3, 3] -1 [2, 2] -2 [1, 1] -3
Then the problem comes. The perceptron learning algorithm is a bug-driven algorithm. Separating the hyperplane is a process of continuous updating. Although the above diagram draws the intermediate process, some superplanes overlap together. What should I do? I thought it was easy at first, and it wouldn’t be a good idea to add a mark to each line. With the above output, you can determine which plane is behind and which plane is behind, so you have the following picture:
As you can see, the markers are also overlapping.
At this time, I found that matplotlib also supports animation, so I am now selling it, with the following story.
Getting started with matplotlib animation
# -*- coding:utf-8 -*- # Filename: sin.py # Author：hankcs # Date: 2015/1/30 22:41 import numpy as np from matplotlib import pyplot as plt from matplotlib import animation # first set up the figure, the axis, and the plot element we want to animate fig = plt.figure() ax = plt.axes(xlim=(0, 2), ylim=(-2, 2)) line, = ax.plot(, , lw=2) # initialization function: plot the background of each frame def init(): line.set_data(, ) return line, # animation function. this is called sequentially def animate(i): x = np.linspace(0, 2, 1000) y = np.sin(2 * np.pi * (x - 0.01 * i)) line.set_data(x, y) return line, # call the animator. blit=true means only re-draw the parts that have changed. anim = animation.FuncAnimation(fig, animate, init_func=init, frames=200, interval=20, blit=True) plt.show()
First create a new picture, coordinates and a blank line as global variables. Then the init method is an initialization method, doing nothing. The parameter i in the animate method represents the current number of frames, and the set of coordinates is generated by accepting i through the sine function and updated into the line. Next, the new anim object is created. The names of several parameters are well understood. The last blit method tells matplotlib to remember to erase the primitives returned by the init method before each frame.
The operation effect is as follows:
Maybe you will wonder how the picture is coming, use other tools to take a screenshot? No, in the face of omnipotent Python, exporting gif is a piece of cake.
ImageMagick is a tool similar to the encoder, download address: http://www.imagemagick.org/script/binary-releases.php
First look at where your configuration files are placed:
import matplotlib matplotlib.matplotlib_fname()
Will output a similar path, open the file with a text file, edit the end:
animation.convert_path: '"C:\Program Files\ImageMagick-6.9.0-Q16\convert.exe"'
Remember to uncomment the comment before “animation.convert_path”. This should be configured, and then you can export the gif in one sentence:
anim.save('perceptron.gif', fps=2, writer='imagemagick')
On my machine, an error occurred unexpectedly:
UserWarning: imagemagick MovieWriter unavailable warnings.warn ("% s MovieWriter unavailable" writer%)
Later, I took a step and found that the logic of matplotlib parsing the configuration file has a bug:
As shown in the figure, the correct location is C:\Program Files\ImageMagick-6.9.0-Q16\convert.exe, but there is an extra number before and after parsing. My solution is to edit C:\Program Files\Python27\Lib\site-packages\matplotlib\__init__.py and add a sentence on line 1101:
rcParams['animation.convert_path'] = 'C:\Program Files\ImageMagick-6.9.0-Q16\convert.exe'
Forcibly overwrite the wrong configuration.
Then it will work fine.
Perceptron gradient descent algorithm visualization
Perceptron algorithm code
Finally, at the most exciting moment, with this knowledge, you can perfectly visualize this simple algorithm:
# -*- coding:utf-8 -*- # Filename: train2.1.py # Author：hankcs # Date: 2015/1/30 16:29 import copy from matplotlib import pyplot as plt from matplotlib import animation training_set = [[(3, 3), 1], [(4, 3), 1], [(1, 1), -1]] w = [0, 0] b = 0 history =  def update(item): """ update parameters using stochastic gradient descent :param item: an item which is classified into wrong class :return: nothing """ global w, b, history w += 1 * item * item w += 1 * item * item b += 1 * item print w, b history.append([copy.copy(w), b]) # you can uncomment this line to check the process of stochastic gradient descent def cal(item): """ calculate the functional distance between 'item' an the dicision surface. output yi(w*xi+b). :param item: :return: """ res = 0 for i in range(len(item)): res += item[i] * w[i] res += b res *= item return res def check(): """ check if the hyperplane can classify the examples correctly :return: true if it can """ flag = False for item in training_set: if cal(item) <= 0: flag = True update(item) # draw a graph to show the process if not flag: print "RESULT: w: " + str(w) + " b: " + str(b) return flag if __name__ == "__main__": for i in range(1000): if not check(): break # first set up the figure, the axis, and the plot element we want to animate fig = plt.figure() ax = plt.axes(xlim=(0, 2), ylim=(-2, 2)) line, = ax.plot(, , 'g', lw=2) label = ax.text(, , '') # initialization function: plot the background of each frame def init(): line.set_data(, ) x, y, x_, y_ = , , ,  for p in training_set: if p > 0: x.append(p) y.append(p) else: x_.append(p) y_.append(p) plt.plot(x, y, 'bo', x_, y_, 'rx') plt.axis([-6, 6, -6, 6]) plt.grid(True) plt.xlabel('x') plt.ylabel('y') plt.title('Perceptron Algorithm (www.hankcs.com)') return line, label # animation function. this is called sequentially def animate(i): global history, ax, line, label w = history[i] b = history[i] if w == 0: return line, label x1 = -7 y1 = -(b + w * x1) / w x2 = 7 y2 = -(b + w * x2) / w line.set_data([x1, x2], [y1, y2]) x1 = 0 y1 = -(b + w * x1) / w label.set_text(history[i]) label.set_position([x1, y1]) return line, label # call the animator. blit=true means only re-draw the parts that have changed. print history anim = animation.FuncAnimation(fig, animate, init_func=init, frames=len(history), interval=1000, repeat=True, blit=True) plt.show() anim.save('perceptron.gif', fps=2, writer='imagemagick')
It can be seen that the hyperplane is attracted by the misclassification point and moves toward it, so that the distance between the two is gradually reduced until it is correctly classified. Through this animation, is there a more intuitive understanding of the gradient reduction algorithm of the perceptron?