Home on Yoke Keong

Post Jupyter Notebooks to WordPress using plugin Documents from Git

Wed, 25 Nov 2020 12:00:00 +0000

Jupyter Notebooks are a great way to communicate your findings or demonstrating certain concepts or applications in the Data Science and Machine Learning world. While you can easily convert notebooks to static pages using nbconvert, there are challenges integrating them with existing publishing platforms like WordPress.

If you are just starting out or don’t mind migrating to another platform, there are awesome alternatives such as FastPages (Github and demo site) by Jeremy Howard and Hamel Husain.

Resources for the Google Cloud Professional Machine Learning Engineer Certification

Sat, 21 Nov 2020 14:24:12 +0000

Recently I passed the Google Cloud Professional ML Engineer Certification Exam and received a few queries on how to prepare for the exam. Hence I am writing up a post to share some free/low-cost training offers and resources that have helped me pass the exam.

One thing to note is that I have previously completed the Google Cloud Professional Data Engineer Certification and have some prior experience with using Google Cloud Platform and TensorFlow/Keras before attempting this certification exam. I shall suggest some resources for learning the portions related to Data Engineering and TensorFlow if you are not familiar with them.

Note on tracking dotfiles across multiple instances easily with git

Tue, 20 Aug 2019 12:00:03 +0000

Came across fast.ai’s dotfiles repo, which provided gems of insight into managing dotfiles in Linux (and Windows WSL) environment by using git bare repos. The repo also provided a very good quick start for some common command line configs. When you are working on various compute instances on cloud and/or on local, it is very handy to track and transfer dotfiles easily.

From scratch

Create an empty repo on Github/Bitbucket/Gitlab named dotfiles

Dim Sum Classifier – from Data to App part 2

Thu, 15 Aug 2019 12:00:52 +0000

Picture Credits here

In the previous post, we see how we can acquire data, process, clean and train an Image Classifier to identify some yummy dim sums.

In this post, we shall look at completing the loop by developing the web app using starlette (a framework similar to that of flask but supports asynchronous IO), setting up and automating deployment of our web app with Github, Docker contanier and Render.

The very helpful fast.ai course team and community has given us a quick start with the following resources:

Dim Sum Classifier - from Data to App part 1

Tue, 13 Aug 2019 12:00:07 +0000

Picture Credits here

In a typically machine learning lifecycle, we will need to acquire data, process data, train and validate/test models and finally deploy the trained models in applications/services. In this first part of two post, inspired by fast.ai 2019 lesson 2, we shall build a Dim Sum (a Cantonese bite-size style of cuisine with many yummy choices) classifier application by leveraging on Google Images as a data source.

Due to the wide variety of choices, we shall focus on 5 common dim sum dishes below, with links for your interest:

Rock, paper, scissors - vision transfer learning with fast.ai

Wed, 07 Aug 2019 12:00:00 +0000

Picture Credits: Wikipedia

In the previous post, we used the Rock, Paper Scissors notebook that trained a custom image classification model from scratch.

While the notebook is demonstrates building custom layers, for such a task, we can also leverage on Transfer Learning using models trained on similar image classification tasks that can often reduce time in training and experimentation and yet achieve results fairly good results, which will be shown here using the fastai v1 library as demonstrated by Jeremy Howard in his awesome Practical Deep Learning for Coders 2019 course.

Serving Rock, Paper, Scissors Image Classifier App built with Tensorflow 2, Keras and Flask

Mon, 05 Aug 2019 12:00:00 +0000

Picture Credits: Pixabay

In this post, we shall be looking at serving a Tensorflow 2 Keras image classification model with a Flask app.

We shall be leveraging on the Rock Paper Scissors Tensorflow 2 Notebook created by Laurence Moroney and built on the Image Classifier App template provided by Fing on the Github repository here.

Training and saving the Model in Google Colab

To train the model, we can run the aforementioned Jupyter Notebook on Google Colab. To train the model successfully, we will need to ensure that Tensorflow 2 beta is installed with the following command:

Find difference between CSV files using PowerShell

Thu, 16 Feb 2017 11:00:08 +0000

If you are on Windows, you can use PowerShell to find the differences between two CSV files. Sample code below:

1$file1 = import-csv -Path "D:\path\to\file1.csv"
2$file2 = import-csv -Path "D:\path\to\file2.csv"
3Compare-Object $file1 $file2 -property column_to_identify_row -IncludeEqual

You can refer to this article for more details.

Activating the free Let's Encrypt SSL certificate for your WordPress site

Thu, 09 Feb 2017 11:00:06 +0000

WPBeginner has a easy to follow tutorial on activating the free Let’s Encrypt SSL certificate for your WordPress site, assuming your web host supports them. The really simple ssl plugin mentioned makes the process painless for existing sites.

If you are on HawkHost, you can view their announcements here on their Let’s Encrypt and 2FA support. In short, you can get the free SSL certificate by:

Logging to cPanel, locate the Security section and click on “Lets Encrypt™ SSL” option to start the process.
You can then select the domain that you would like HawkHost to issue the SSL certificate to.
You will be brought to a confirmation page. On confirmation, HawkHost will generate the SSL certificate for you.
Back in your WordPress admin page, activate the really simple ssl plugin and click on the button “Go ahead, activate SSL!”.
You might be logged out of the admin page. If so, login again and you will discover that your WordPress site is now on SSL.
Refer back to the WPBeginner tutorial for additional details like Google Analytics updates.

Find version of python package installed

Thu, 04 Aug 2016 11:00:52 +0000

Below are 3 methods we can try to find the version of an installed python package. We shall use scipy as an example.

Using pip

Method 1 - For pip 1.3 and above: pip show scipy

Method 2 - Alternative (works with older versions of pip): pip freeze | grep scipy

Using version attribute

Method 3 - Launch python/ipython, then execute the commands below:

1import scipy
2scipy.__version__

Reference for Method 1 is here. Reference for Method 2 is here.

Tool to read large text files

Thu, 21 Jul 2016 11:00:05 +0000

Sometimes, you might need to view and search very large text files like logs or SQL dumps. On Windows and Linux, you can give glogg a try. Please note that this application is read-only.

Running Apache Spark with sparklyr and R in Windows

Wed, 06 Jul 2016 17:00:50 +0000

RStudio recently released the sparklyr package that allows users to connect to Apache Spark instances from R. In addition, this package offers dplyr integration, allowing you to utilize Spark as you use dplyr functions like filter and select, which is very convenient. The package will also assist you in downloading and installing Apache Spark if it is a fresh install. This post covers the local install of Apache Spark via sparklyr and RStudio in Windows 10.

Downloading R packages securely

Tue, 25 Aug 2015 11:00:52 +0000

JJ Allaire from RStudio wrote a step-by-step guide on how to configure secured downloads of R packages. Secured downloading of R packages (via HTTPS connection) ensures that you get your packages from legitimate, trusted sources.

Launching matlab command line in Windows

Tue, 21 Jul 2015 11:00:28 +0000

Sometimes you would only want to launch Matlab’s command line window instead of the full IDE. To do that in Windows, type the following command in the command prompt: matlab -nodesktop

Getting session information in Python

Tue, 07 Jul 2015 11:00:08 +0000

We’ve gone through how to get session information in R previously, so how do we do the same for Python? It seems that there is no single convenient function available so here’s one approach.

To get the system information, you can utilize the commonly used IPython package:

1import IPython
2IPython.sys_info()

To find out packages that have been loaded at the time (includes modules loaded by Python itself and by any Python IDE), you can utilize the sys.modules.keys() method. The code below gets the package name rather than the sub-components.

Getting session information in R

Tue, 30 Jun 2015 11:00:15 +0000

When troubleshooting R bugs or asking for assistance in mailing lists and sites like StackOverflow, it is good to review or present information about your system and packages loaded.

I much prefer the session_info() function from the devtools package over the default sessionInfo() function as it’s output is not only more readable, it also provides useful information like timezone and additional packages (non-base) loaded at the time.

Assuming you have the devtools packages already installed, you can invoke the function in one line:

Updating packages after R upgrade

Tue, 26 May 2015 11:00:50 +0000

Note to self: After upgrading R (or Revolution R Open) on Windows, run the following command to update the packages at one go.

1update.packages(checkBuilt = TRUE, ask = FALSE)

Drawing map shapes with rgdal

Tue, 05 May 2015 11:00:06 +0000

In this post, let us explore the R package rgdal for map shape plotting. We shall attempt to plot the map of Singapore and display major road networks in Singapore.

Pre-requisites

To get the data you need, you can go to a GIS provider. In this post, we shall be using diva-gis.

Steps:

1. Go to: http://www.diva-gis.org/gdata

2. Select ‘Spatial Data Download’

3. Select Country = ‘Singapore’

4. Select Subject = ‘Administrative areas (GADM)’

Reading tabular data with readr package

Tue, 14 Apr 2015 11:00:57 +0000

Recently, Hadley Wickham introduced a new package to read tabular data (such as CSV), lines and entire files.

Advantages include:

1. Helpful defaults over base R read.csv such as: Characters are never automatically converted to factors and row names are never set.

2. Faster reads.

3. When reading large files, a progress bar is displayed.

4. A very useful problems() function that allows you to zoom into rows of data that readr has issues loading into for e.g. expected integer for this column but actual data is text.

Handling line endings on Windows with git

Tue, 07 Apr 2015 11:00:27 +0000

Note to self: When collaborating on different platforms, one of the most common issue is line endings - LF on Mac/Linux and CRLF on Windows. With git, you can address this issue in the following ways:

1) Configure Global Settings

Windows users: git config --global core.autocrlf true
Mac/Linux users: git config --global core.autocrlf input

2) Configure Per-repository settings

1echo "* text=auto" > .gitattributes
2git add .
3git commit -m "Files play nice in Mac, Linux and Windows"
4git push origin master

Additional reading: StackOverflow thread Github help

Reading excel files with readxl R package

Tue, 31 Mar 2015 11:00:05 +0000

Recently, Hadley Wickham introduced a new package to read Excel files (XLS, XLSX). The main advantage is that no external dependencies is required for readxl. (xlsx package requires Java Runtime to be installed)

With xlsx:

1library(xlsx)
2library(httr)
3url <- "https://rawgit.com/yoke2/dsxref/master/iris.xlsx"
4GET(url, write_disk("iris.xlsx", overwrite=TRUE))
5iris <- read.xlsx("iris.xlsx", sheetIndex=1, header=TRUE)
6head(iris, 3)

1## NA. Sepal.Length Sepal.Width Petal.Length Petal.Width Species
2## 1 1 5.1 3.5 1.4 0.2 setosa
3## 2 2 4.9 3.0 1.4 0.2 setosa
4## 3 3 4.7 3.2 1.3 0.2 setosa

With readxl:

Download files over HTTPS in R with httr

Tue, 10 Mar 2015 11:00:52 +0000

To download a file over HTTP connection, we normally use download.file command in R, for example:

1url = "http://vincentarelbundock.github.io/Rdatasets/csv/datasets/iris.csv"
2download.file(url, "iris.csv", quiet=TRUE)

For HTTPS connections, download.file may give you some issues. In situations like this you can consider using the httr package to download files:

1library(httr)
2url <- "https://rawgit.com/yoke2/dsxref/master/iris.xlsx"
3GET(url, write_disk("iris.xlsx", overwrite=TRUE))

Data Science Learning - A Cross Reference

Tue, 17 Feb 2015 11:00:12 +0000

While learning data science, I’ve discovered that it is very useful to think of the data science processing as a “pipeline” i.e. a series of actions in a process. Along this pipeline, you will be tackling lots of “How do I…” questions like “How do I remove NA values?” and “How do I create N-grams?”

Furthermore, given the many data science tools and languages available online, you will most likely ask the same questions again when you are learning how to perform data science tasks in another language. While Google Search and Stack Overflow/Stack Exchange comes in very handy when searching for answers, I wanted some structure - a collection of sorts - to these questions and have working examples in different language implementations.

Completed Data Science Specialization

Tue, 20 Jan 2015 11:00:32 +0000

“Data Science, a 10-course specialization by Johns Hopkins University on Coursera. Specialization Certificate earned on December 22, 2014”

Finally received the certification!

A virtual environment for data science

Tue, 13 Jan 2015 11:00:23 +0000

I wanted to conveniently use data science tools without the hassle of installing the required languages and packages, while benefiting from the strengths of the Linux command line tools. There is a pre-packaged VM called the Data Science Toolbox that fills this need.

It comes with R and Python installed, along with the respective popular data analysis packages for R and Python. You will be able to install the VM successfully by following the instructions on the website, including installation of pre-requisites like VirtualBox and Vagrant.

Speeding up R

Tue, 16 Dec 2014 11:00:38 +0000

When processing large datasets, some R commands operations requiring linear algebra libraries may run very slowly. As such you might want to try Revolution R Open, an enhanced R distribution using multi-threaded libraries. You can see some benchmarks here and here.

Exploring Focal Length with Exiftool and R

Thu, 16 Oct 2014 12:00:36 +0000

Picture Credits: Pixabay

A good way to understand your shooting style and guide your future camera equipment buying decisions will be to discover your frequently used focal lengths.

Focal Lengths can be extract from photos that have EXIF data, which in short refers to data on how these photos are taken. You can find out more from an introductory article here. In this post, we are going to explore focal length usage with the flexible Exiftool and R.

#DataForGood Projects

Mon, 01 Jan 0001 00:00:00 +0000

#DataForGood Projects

This page lists the various voluntary data projects that I’ve led or contributed to with DataKind Singapore in reverse chronological order.

Volunteer for WaterPoint Data Exchange (2019)

I assisted to create an Image Classification Model to identify WaterPoint images.

Highlights

Multi-class Image Classification
Transfer Learning
Tools used: Python, fastai

Github

Data Ambassador for Projects

Community Justice Centre & Law Society Pro Bono Services (2018)
Singapore Children’s Society (2017)

I co-led the above projects as a Data Ambassador, collaborating with non-profit representatives to discover business needs and translate them into data requirements, and worked with volunteer teams to deliver data insights and proof-of concepts in a weekend DataDive.

AI/ML Projects (Masters)

Mon, 01 Jan 0001 00:00:00 +0000

AI/ML Projects (Masters)

This page show cases work done during my internship as well as the AI and Machine Learning Systems (in reverse chronological order) collaboratively built with my fellow team mates during my study in the Masters of Technology in Intelligent Systems in the National University of Singapore, Institute of System Science from 2019 to 2020.

Internship at the Chope Group

During this internship, I worked with Product & Engineering Teams and Data Science Lead on several projects and productionizing Machine Learning Services end-to-end.

Coursera Data Science Specialization Projects

Mon, 01 Jan 0001 00:00:00 +0000

Coursera Data Science Specialization Projects

This page showcases the Data Science Projects that I’ve completed as part of the Johns Hopkins University Data Science Specialization Track offered through Coursera from 2014 to 2015. You will see a brief description on the project, followed by links to applications, repositories, presentations and/or reports, where applicable.

Capstone Project

In this inaugural capstone run, which is offered in partnership with SwiftKey, I’ve created a text prediction application that allows a user to enter a phrase and subsequently predict the next word the user might enter.

Talks and Micro-Projects

Mon, 01 Jan 0001 00:00:00 +0000

Talks

Building a Food Classifier App End-to-End: Walk-through & Sharings (Jan-2020)

I conducted a hands-on workshop to walk participants through the creation of a food classifier webapp from data collection to application deployment and sharing of lessons learnt.

GitHub

Journal Clubs, Learnings and Walkthroughs

I shared presentations and walkthroughs of learnings from journals/articles in areas of interests in my Github repo. You can access the link below.

GitHub

Micro Projects

Below are some micro projects I’ve written for AI, Machine Learning and Data Science. Please look at my blog for more posts.