Building a Neural Net to Visualize High-Dimensional Data in TensorFlow

Word embeddings and high-dimensional data are ubiquitous in many facets of deep learning research such as natural language processing, building recommendation systems. Google in the recent times has open sourced the embedding project that works as an interactive collaborative visualization tool to visualize the high-dimensional data.  The embedding works as a navigational map from the points to the data inputs in Euclidean space.  Many types of big data require visualization of graphs, neural nets, writing summaries on the parameters of weights, sigmoid activation functions, and the accuracy of the machine learning models.

In this tutorial, we use the MNIST handwritten digits’ dataset. More explanation on the mechanics of the MNIST handwritten digits dataset can be found on Yann Lecun’s website.  As described by Yann Lecun.

“THE IDX FILE FORMAT

the IDX file format is a simple format for vectors and multidimensional matrices of various numerical types.

The basic format is

magic number
size in dimension 0
size in dimension 1
size in dimension 2
…..
size in dimension N
data

The magic number is an integer (MSB first). The first 2 bytes are always 0.

The third byte codes the type of the data:
0x08: unsigned byte
0x09: signed byte
0x0B: short (2 bytes)
0x0C: int (4 bytes)
0x0D: float (4 bytes)
0x0E: double (8 bytes)

The 4-th byte codes the number of dimensions of the vector/matrix: 1 for vectors, 2 for matrices….

The sizes in each dimension are 4-byte integers (MSB first, high endian, like in most non-Intel processors).

The data is stored like in a C array, i.e. the index in the last dimension changes the fastest.”

Several deep learning frameworks have measured the benchmarks leveraging MNIST handwritten digits.  MNIST dataset is easily accessible to TensorFlow framework.  TensorFlow represents the data as tensors in the graph.  Tensors most of the time are representatives of high-dimensional data.  MNIST dataset has around Z784 dimensions. The world we live only has Z3 space. Hence, visualizing the data in 784 dimensions for human eye is a daunting task.  Therefore, various techniques such as Principle Component Analysis or t-distributed Stochastic Neighbor Embedding (t-SN3) can be adapted to reduce the curse of the dimensionality to reduce the dimensions and visualize the data in lower dimensional space interpretable to the human eye.   

Building a Neural Net to visualize the tensors of the hidden layer activation is possible through TensorBoard’s embedded projector.  For this, numpy, matplotlib libraries will be imported.

Figure 1. Importing libraries.

The input data for this example will be leveraged from MNIST dataset of handwritten digits.

Figure 2. Importing Input data from MNIST dataset.

 

Extracting MNIST_data/train-images-idx3-ubyte.gz

Extracting MNIST_data/train-labels-idx1-ubyte.gz

Extracting MNIST_data/t10k-images-idx3-ubyte.gz

Extracting MNIST_data/t10k-labels-idx1-ubyte.gz

Size of:

– Training-set:                           55000

– Test-set:                 10000

– Validation-set:         5000

 

The hyper parameters will be specified externally as constant parameters as these are not learned by the neural networks.

Figure 3. Hyper-parameters.

The graph needs to be constructed here, but a fully convolutional neural network or FC Layer (LeNet) needs to be defined along with weights, and bias variables.

Figure 4. Definition of weights, convolutional neural network, and bias variables.

Since the helper functions have been activated, the graph can now be created:

Figure 5. Creation of graph.

 

An interactive graph session will be launched in the Jupyter notebook after executing the following code:

Figure 6. Launching the graph session.

After executing the code, it displays the training accuracy for the neural network.

Training epoch: 1
iter   0:       Loss=2.28,    Training Accuracy=37.0%
iter 100:       Loss=0.45,    Training Accuracy=85.0%
iter 200:       Loss=0.21,    Training Accuracy=95.0%
iter 300:       Loss=0.34,    Training Accuracy=89.0%
iter 400:       Loss=0.29,    Training Accuracy=90.0%
iter 500:       Loss=0.18,    Training Accuracy=97.0%
---------------------------------------------------------
Epoch: 1, validation loss: 0.21, validation accuracy: 94.1%
---------------------------------------------------------

Training epoch: 2
iter   0:       Loss=0.14,    Training Accuracy=97.0%
iter 100:       Loss=0.19,    Training Accuracy=94.0%
iter 200:       Loss=0.11,    Training Accuracy=95.0%
iter 300:       Loss=0.06,    Training Accuracy=99.0%
iter 400:       Loss=0.08,    Training Accuracy=96.0%
iter 500:       Loss=0.12,    Training Accuracy=97.0%
---------------------------------------------------------
Epoch: 2, validation loss: 0.14, validation accuracy: 96.0%
---------------------------------------------------------

Training epoch: 3
iter   0:       Loss=0.16,    Training Accuracy=96.0%
iter 100:       Loss=0.15,    Training Accuracy=96.0%
iter 200:       Loss=0.08,    Training Accuracy=97.0%
iter 300:       Loss=0.11,    Training Accuracy=94.0%
iter 400:       Loss=0.10,    Training Accuracy=96.0%
iter 500:       Loss=0.09,    Training Accuracy=97.0%
---------------------------------------------------------
Epoch: 3, validation loss: 0.11, validation accuracy: 96.7%
---------------------------------------------------------

Training epoch: 4
iter   0:       Loss=0.03,    Training Accuracy=100.0%
iter 100:       Loss=0.04,    Training Accuracy=99.0%
iter 200:       Loss=0.11,    Training Accuracy=94.0%
iter 300:       Loss=0.05,    Training Accuracy=98.0%
iter 400:       Loss=0.11,    Training Accuracy=97.0%
iter 500:       Loss=0.11,    Training Accuracy=95.0%
---------------------------------------------------------
Epoch: 4, validation loss: 0.09, validation accuracy: 97.2%
---------------------------------------------------------

Training epoch: 5
iter   0:       Loss=0.11,    Training Accuracy=97.0%
iter 100:       Loss=0.04,    Training Accuracy=98.0%
iter 200:       Loss=0.04,    Training Accuracy=100.0%
iter 300:       Loss=0.02,    Training Accuracy=100.0%
iter 400:       Loss=0.11,    Training Accuracy=97.0%
iter 500:       Loss=0.04,    Training Accuracy=99.0%
---------------------------------------------------------
Epoch: 5, validation loss: 0.08, validation accuracy: 97.7%
---------------------------------------------------------

Training epoch: 6
iter   0:       Loss=0.04,    Training Accuracy=100.0%
iter 100:       Loss=0.07,    Training Accuracy=98.0%
iter 200:       Loss=0.08,    Training Accuracy=98.0%
iter 300:       Loss=0.04,    Training Accuracy=99.0%
iter 400:       Loss=0.04,    Training Accuracy=99.0%
iter 500:       Loss=0.05,    Training Accuracy=97.0%
---------------------------------------------------------
Epoch: 6, validation loss: 0.07, validation accuracy: 97.9%
---------------------------------------------------------
Training epoch: 7
iter   0:       Loss=0.04,    Training Accuracy=99.0%
iter 100:       Loss=0.03,    Training Accuracy=99.0%
iter 200:       Loss=0.07,    Training Accuracy=99.0%
iter 300:       Loss=0.06,    Training Accuracy=98.0%
iter 400:       Loss=0.13,    Training Accuracy=96.0%
iter 500:       Loss=0.07,    Training Accuracy=97.0%
---------------------------------------------------------
Epoch: 7, validation loss: 0.07, validation accuracy: 97.8%
---------------------------------------------------------
Training epoch: 8
iter   0:       Loss=0.03,    Training Accuracy=99.0%
iter 100:       Loss=0.01,    Training Accuracy=100.0%
iter 200:       Loss=0.03,    Training Accuracy=99.0%
iter 300:       Loss=0.02,    Training Accuracy=100.0%
iter 400:       Loss=0.02,    Training Accuracy=100.0%
iter 500:       Loss=0.05,    Training Accuracy=98.0%
---------------------------------------------------------
Epoch: 8, validation loss: 0.07, validation accuracy: 97.8%
---------------------------------------------------------

Training epoch: 9
iter   0:       Loss=0.04,    Training Accuracy=99.0%
iter 100:       Loss=0.01,    Training Accuracy=100.0%
iter 200:       Loss=0.07,    Training Accuracy=98.0%
iter 300:       Loss=0.03,    Training Accuracy=99.0%
iter 400:       Loss=0.03,    Training Accuracy=99.0%
iter 500:       Loss=0.09,    Training Accuracy=97.0%
---------------------------------------------------------
Epoch: 9, validation loss: 0.07, validation accuracy: 97.9%
---------------------------------------------------------

Training epoch: 10
iter   0:       Loss=0.02,    Training Accuracy=100.0%
iter 100:       Loss=0.03,    Training Accuracy=100.0%
iter 200:       Loss=0.02,    Training Accuracy=100.0%
iter 300:       Loss=0.02,    Training Accuracy=100.0%
iter 400:       Loss=0.01,    Training Accuracy=100.0%
iter 500:       Loss=0.01,    Training Accuracy=100.0%
---------------------------------------------------------
Epoch: 10, validation loss: 0.07, validation accuracy: 98.1%
---------------------------------------------------------

 

Considering the dataset has around 10,000 samples, the hidden layer in the neural network has around 200 nodes.  The tensor output data has around 10,000 x 200 layers.

Figure 7. Assigning of hidden layer of the convolutional neural network to the variables.

The objective of this code is to visualize the high-dimensional data in a low dimensional way to make it interpretable for the human eye.  The metadata can be downloaded into metadata.tsv from the TensorBoard that can store the index and label for individual samples.  This file has to be added to the points embedding config. File.

 

Figure 8. Creating config. object.

Running the session in Jupyter Notebook will evaluate the tensor for the embeddings and save the data.

Figure 9. It creates the tensor in the deep learning model cockpit configuration log file.

I have shared the code on GitHub at GPSingularity:

Once the session is executed, it can now be visualized in TensorBoard with the following command:

 

tensorboard –logdir=logs/embedding/ –host localhost The data will be displayed as follows in TensorBoard.  Results from the program

Figure 10. TensorBoard Embedded Projector to visualize High-dimensional data in low dimension.

This program I’ve written is inspired by Easy TensorFlow http://www.easy-tensorflow.com/’sembedded visualization.

 

I’ve shared my program in GPSingularity Github.

 

References

Easy TensorFlow (2018). Easy TensorFlow. Retrieved from http://www.easy-tensorflow.com/

Smilkov, D., Thorat, N., Nicholson, C., Reif, E., Viegas, F. B., & Watternberg, M. (2016). Embedding Projector: Interactive Visualization and Interpretation of Embeddings. Retrieved from https://arxiv.org/pdf/1611.05469v1.pdf

26 thoughts on “Building a Neural Net to Visualize High-Dimensional Data in TensorFlow

  1. nmd Posted On

    I found your blog website on google and verify just a few of your early posts. Proceed to keep up the excellent operate. I just further up your RSS feed to my MSN News Reader. Searching for ahead to reading extra from you afterward!?

    Reply
  2. yeezy boost 350 Posted On

    I want to express my passion for your generosity in support of all those that should have help on this particular subject matter. Your personal dedication to passing the solution all through ended up being extremely insightful and have usually empowered women just like me to reach their targets. This warm and helpful advice signifies a whole lot a person like me and extremely more to my peers. Warm regards; from each one of us.

    Reply
  3. yeezy shoes Posted On

    I wish to express some thanks to you just for bailing me out of such a situation. As a result of looking out through the search engines and meeting concepts that were not pleasant, I thought my entire life was gone. Living devoid of the answers to the difficulties you’ve sorted out by means of your good report is a serious case, as well as those that would have in a negative way affected my career if I hadn’t encountered the website. Your actual talents and kindness in maneuvering all areas was very helpful. I’m not sure what I would’ve done if I hadn’t encountered such a subject like this. I’m able to at this moment look forward to my future. Thanks for your time so much for this specialized and result oriented guide. I won’t hesitate to refer your web blog to anyone who would like guidelines on this area.

    Reply
  4. golden goose sneakers Posted On

    I must express some thanks to the writer for bailing me out of this trouble. Just after searching throughout the world wide web and seeing tricks that were not powerful, I assumed my life was done. Being alive devoid of the answers to the difficulties you’ve solved through your main post is a crucial case, and the kind which might have negatively affected my entire career if I hadn’t come across the blog. Your own personal capability and kindness in dealing with the whole lot was crucial. I don’t know what I would’ve done if I had not come upon such a solution like this. I can now relish my future. Thank you so much for the high quality and effective help. I will not think twice to propose your blog to any individual who ought to have assistance about this subject.

    Reply
  5. Adidas NMD Runner men women white Posted On

    I discovered your weblog website on google and check a few of your early posts. Proceed to maintain up the very good operate. I simply extra up your RSS feed to my MSN Information Reader. In search of forward to reading extra from you later on!?

    Reply
  6. hermes birkin Posted On

    Thank you so much for giving everyone such a pleasant possiblity to check tips from this web site. It is usually so beneficial plus full of a great time for me personally and my office peers to search the blog not less than three times per week to find out the new stuff you have. And of course, I am just always astounded with your awesome tricks you give. Certain 4 facts in this article are indeed the most suitable I’ve ever had.

    Reply
  7. nike free run Posted On

    After examine a number of of the blog posts on your web site now, and I really like your way of blogging. I bookmarked it to my bookmark web site checklist and shall be checking back soon. Pls take a look at my web page as properly and let me know what you think.

    Reply
  8. nmd Posted On

    I not to mention my guys ended up going through the excellent points located on the blog then all of a sudden developed a terrible suspicion I had not expressed respect to the website owner for those secrets. The men happened to be totally glad to read them and have in effect in reality been loving these things. Appreciation for truly being well considerate and then for picking out these kinds of superb information millions of individuals are really wanting to discover. My personal honest regret for not saying thanks to you earlier.

    Reply
  9. cheap nfl jerseys Posted On

    After I originally commented I clicked the -Notify me when new comments are added- checkbox and now each time a remark is added I get 4 emails with the identical comment. Is there any method you possibly can remove me from that service? Thanks!

    Reply
  10. adidas nmd Posted On

    I as well as my friends were examining the great key points on your web site and so the sudden I had a horrible suspicion I never expressed respect to you for those techniques. All the people happened to be totally excited to read through them and have in effect simply been enjoying those things. Thanks for turning out to be well helpful and also for opting for variety of outstanding areas millions of individuals are really eager to understand about. My very own sincere apologies for not expressing gratitude to you sooner.

    Reply
  11. yeezy boost Posted On

    Thanks so much for giving everyone a very superb chance to read from this site. It really is so excellent and as well , packed with amusement for me personally and my office co-workers to search your website no less than three times in one week to read the newest items you have. And indeed, I am just always pleased with your gorgeous guidelines you give. Certain two points on this page are completely the most suitable we’ve ever had.

    Reply
  12. james harden shoes Posted On

    Oh my goodness! an incredible article dude. Thanks Nevertheless I am experiencing issue with ur rss . Don抰 know why Unable to subscribe to it. Is there anyone getting similar rss problem? Anybody who is aware of kindly respond. Thnkx

    Reply
  13. nmd uk Posted On

    Needed to draft you a very small word just to thank you once again just for the pleasant solutions you have shared here. It is quite tremendously generous with people like you to present openly just what a number of us might have distributed as an electronic book to earn some profit for themselves, most importantly considering the fact that you might have done it in case you desired. The techniques in addition served like the fantastic way to recognize that other people have the same desire much like my own to grasp a great deal more concerning this condition. I’m certain there are millions of more pleasurable sessions ahead for those who view your site.

    Reply
  14. Adidas NMD Runner White Stripes Posted On

    I must express my love for your kind-heartedness giving support to women who really need help on this important study. Your special commitment to passing the solution all over appeared to be incredibly invaluable and have surely encouraged guys like me to attain their dreams. The interesting guidelines denotes a great deal a person like me and extremely more to my mates. Thanks a ton; from everyone of us.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *