Dong Liang's Blog

Sample post

Sat, 30 Nov 2019 00:00:00 +0000

Consectetur adipiscing elit. Donec a diam lectus. Sed sit amet ipsum mauris. Maecenas congue ligula ac quam viverra nec consectetur ante hendrerit. Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean ut gravida lorem.

Consectetur adipiscing elit
Donec a diam lectus
Sed sit amet ipsum mauris

Ut turpis felis, pulvinar a semper sed, adipiscing id dolor. Pellentesque auctor nisi id magna consequat sagittis. Curabitur dapibus enim sit amet elit pharetra tincidunt feugiat nisl imperdiet. Ut convallis libero in urna ultrices accumsan. Donec sed odio eros. Donec viverra mi quis quam pulvinar at malesuada arcu rhoncus. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. In rutrum accumsan ultricies. Mauris vitae nisi at sem facilisis semper ac in est.

Nunc diam velit, adipiscing ut tristique vitae, sagittis vel odio. Maecenas convallis ullamcorper ultricies. Curabitur ornare, ligula semper consectetur sagittis, nisi diam iaculis velit, id fringilla sem nunc vel mi. Nam dictum, odio nec pretium volutpat, arcu ante placerat erat, non tristique elit urna et turpis. Quisque mi metus, ornare sit amet fermentum et, tincidunt et orci. Fusce eget orci a orci congue vestibulum.

Ut dolor diam, elementum et vestibulum eu, porttitor vel elit. Curabitur venenatis pulvinar tellus gravida ornare. Sed et erat faucibus nunc euismod ultricies ut id justo. Nullam cursus suscipit nisi, et ultrices justo sodales nec. Fusce venenatis facilisis lectus ac semper. Aliquam at massa ipsum. Quisque bibendum purus convallis nulla ultrices ultricies. Nullam aliquam, mi eu aliquam tincidunt, purus velit laoreet tortor, viverra pretium nisi quam vitae mi. Fusce vel volutpat elit. Nam sagittis nisi dui.

Suspendisse lectus leo, consectetur in tempor sit amet, placerat quis neque

Etiam luctus porttitor lorem, sed suscipit est rutrum non. Curabitur lobortis nisl a enim congue semper. Aenean commodo ultrices imperdiet. Vestibulum ut justo vel sapien venenatis tincidunt.

Phasellus eget dolor sit amet ipsum dapibus condimentum vitae quis lectus. Aliquam ut massa in turpis dapibus convallis. Praesent elit lacus, vestibulum at malesuada et, ornare et est. Ut augue nunc, sodales ut euismod non, adipiscing vitae orci. Mauris ut placerat justo. Mauris in ultricies enim. Quisque nec est eleifend nulla ultrices egestas quis ut quam. Donec sollicitudin lectus a mauris pulvinar id aliquam urna cursus. Cras quis ligula sem, vel elementum mi. Phasellus non ullamcorper urna.

Unsupervised Pre-training of Image Features on Non-Curated Data

Sun, 10 Nov 2019 00:00:00 +0000

Unsupervised Pre-training of Image Features on Non-Curated Data

Code && Paper

名词解释

curated-data：整理过后的数据，一般不会存在类别不平衡现象，例如ImageNet，没有长尾分布。
如果按照监督信息的强弱排名的话：标记数据>无标记数据（curated data）>无标签数据（non-curated data）

问题定义

如何利用未整理的数据，训练一个pre-trained的模型参数，从而提高下游任务的性能？

Self-supervised learning：利用另一个模型或者某些规则生成一个伪label(pseudo label)，基于这些pseudo label去优化我们的target network。 $y_n$ 代表的就是伪label $min_{\theta , W}\frac{1}{N}\sum_{n=1}^N{l(g_W(f_{\theta}(x_n),y_n)}\\$
Clustering：利用聚类算法的思想，生成label或者对应的feature representation。 $z_n$ 代表的是cluster的assignment。 $min_{\theta , W}\frac{1}{N}\sum_{n=1}^N{l(g_W(f_{\theta}(x_n),z_n)}\\ min_{C \in R^{dxk}}\frac{1}{N}\sum_{n=1}^N{min_{z_n\in\{0,1\}^k}||f_{\theta}(x_n)-C_{z_n}||_2^2}$
对于self-supervised learning的方法来说，他对于数据的分布是相对稳定的，但是他不能捕捉图像之间的统计信息。但是对于Deep Clustering的方法来说，他对于数据的分布是不稳定的，不同的数据分布会导致不同的聚类效果。但是他可以考虑图像之间的统计信息。如下图所示

Method	inter-image statistics	stable to distribution change
Self-Sup	N	Y
Deep Clustering	Y	N

DeeperClustering

idea

上述我们分别介绍了Self-supervised 和 Clustering方法的优缺点，那么我们能不能考虑将两个方法结合起来呢。一种最直观的解决方案就是将两者的loss相加，但是作者认为如果相加的话，默认两个任务是相对独立的。为了让其有更多的相互影响，作者采用了笛卡尔乘积来将两者结合： $min_{\theta , W}\frac{1}{N}\sum_{n=1}^N{l(g_W(f_{\theta}(x_n),y_n \bigotimes z_n)}\\$ 也就是将两者concat。

但是上述的笛卡尔乘积方式当某一个维度很长的时候，会造成latent space很大，所以为了解决该问题，作者提出了分层的DeeperClustering方法。

overview

网络结构如下所示，整个训练过程迭代的通过two-step来完成。

第一步：通过cluster生成pseudo label。这个过程是分级的，采用两级结构。首先我们通过kmeans对conv输出的特征进行聚类，聚成m类。然后我们利用self-sup的思想，对每个图像进行旋转0，90，180，270度，所以第一层在训练过程中有4m个class。我们再对每一个cluster内部再次进行聚类，聚类成K类。所以一共的类别是m*k类。
第二步：我们再通过生成的pseudo label进行学习与训练。
Loss function

loss函数定义如下所示。总体上来说有两部分组成。
$l(Vf_{\theta }(x_n), y_n)$ 代表的是第一级别的loss函数。一共是4m类，不仅仅有原来的m类，还有旋转后的，所以一共是4m类。
$\sum_{s=1}^S{y_{ns}l(W_sf_{\theta }(x_n),z_n^s)}$ 代表的是第二级的loss。针对上述的4m类，每一类都有一个分类器，我们计算他的输出和二级聚类的label的loss。

$\frac{1}{N}\sum_{n=1}^N{[l(Vf_{\theta }(x_n), y_n) + \sum_{s=1}^S{y_{ns}l(W_sf_{\theta }(x_n),z_n^s)}]}$

Invariant Information Clustering for Unsupervised Image Classification and Segmentation

Sat, 09 Nov 2019 00:00:00 +0000

Invariant Information Clustering for Unsupervised Image Classification and Segmentation

Code && Paper

问题定义

如何基于无标签的数据，从头训练一个神经网络分类器？

主要思想

利用paired的数据，通过最大化他们之间的互信息来优化整个分类器网络。
最大化互信息所代表的物理意义最大化已知一个数据的分布情况下，另一个数据分布减少的不确定性。也就是两个数据的相关性强。
如何生成paired的数据呢？通过augmentation的方式可以生成paired的数据。

IIC

overview

网络结构如下所示，optional clustering代表的是可选的clustering。比如说，我们原有的只需要聚类10类，我们通过optional clustering可以选择聚100类。避免noise data的影响。

Loss function

我们知道互信息的定义为 $I(X,Y)=\sum_{x \in X}\sum_{y \in Y}{p(x,y)log{\frac{p(x, y)}{p(x)p(y)}}}$ ，所以我们需要分别计算关于X和Y的边缘概率分布和联合概率分布。为了计算这两个分布，我们在batch范围内计算一个矩阵P， $P=\frac{1}{n}\sum_{i=1}^n{\Phi (x_i)\Phi ({x_i}^{'})^T}$ ，其中 $\Phi (x_i)$ 代表的是分类器的输出，即就是每个class的概率。在此基础上，我们可以计算I(z,z’), $I(z,z')=\sum_{c=1}^C{\sum_{c'=1}^C{P_{cc'}ln{\frac{P_{cc'}}{P_cP_{c'}}}}}$
物理意义：互信息可以写成熵减去条件熵的形式： $I(z, z')=H(z)-H(z\|z')$ 。最大化互信息，就代表着最大化熵，同时也代表着最小化条件熵。前者让每个cluster的概率尽可能相等，后者想要突出某个cluster的概率。所以有点对抗训练的意思。

为什么L1正则项产生稀疏的权重，L2正则项产生相对平滑的权重

Tue, 10 Sep 2019 00:00:00 +0000

L1和L2正则化的区别

L1 和L2正则项的定义如下： $L1 = \sum_{i} |w_i|\\ L2 = \sum_{i} (w_i)^2$
首先我们先计算一下他们对应的导数，导入如下所示： $\frac{\partial L1}{\partial w_i} = 1 or -1 \rightarrow w^{t+1}_i = w^t_i + \eta {(-1 or 1)}\\ \frac{\partial L2}{\partial w_i} = w_i \rightarrow w^{t+1}_i = w^t_i + \eta w_i$
所以我们看到L1每次更新的时候会更新一个定值，那么若干次迭代之后，权重就有可能减少为0。但是L2每个更新的时候更新的值的大小和$w_i$ 的值是有关系的。当$w_i$ 趋近与0时，那么对应的导数值也会更新，所以他会不停的接近0，但并不会是0。此外，我们还可以得到，L2相对L1更稳定一些。
L1 产生0的权重也可以起到特征选择的作用，假设我们有$X_0…X_i…X_n$ n个特征，通过分配不同的权重$w_0…w_i…w_n$，然后使用L1 来做特征选择。因为我们使用L1正则，会比较偏向于将某些w设置为0，从而达到了特征选择的作用。
L2 可以迅速产生接近0的权值，但并不是0，所以会比较平滑。
此外，我们还可以从几何的角度来理解。
- 假设我们的Loss函数是$(y - wx)^2$, 那么我们的几何解释如下图所示：
- 其中左图表示L1，右图表示L2。绿色代表的是loss的等高线，$w_1, w_2$ 在L1中的取值空间如左图的菱形所示。在L2中的取值空间如右图的圆形所示。从等高线和取值空间的交点可以看到L1更容易倾向一个权重偏大一个权重为0。L2更容易倾向权重都较小。
- 为什么等高线是类似圆形？以linear regression举例，我们的目标函数$E(w) = (y - w^Tx)^2$，可以看到E是关于w的平方函数，所以是类似圆形(椭圆形)。
主要参考

Color Post

Sat, 18 May 2019 00:00:00 +0000

What a colorful post!

This is an idea that came from xukimseven/HardCandy-Jekyll looking at this cheerful and colorful them, I wanted to enable something similar for mine.

You can go fork and star hers too! 😉

How does it work?

Basically you need to add just one thing, the color:

---
layout: post
title: Color Post
color: brown
---

It can either be a html color like brown (which look like red to me). Or with the rgb:

---
layout: post
title: Color Post
color: rgb(165,42,42)
---

The background used is lineart.png from xukimseven you can edit it in the config file. If you want another one, put it in /assets/img as well.

⚠️ It’s a bit hacking the css in the post.html

Feature images

Sat, 29 Nov 2014 00:00:00 +0000

This is an example of a post which includes a feature image specified in the front matter of the post. The feature image spans the full-width of the page, and is shown with the title on permalink pages.

Markdown and HTML

Fri, 28 Nov 2014 00:00:00 +0000

Jekyll supports the use of Markdown with inline HTML tags which makes it easier to quickly write posts with Jekyll, without having to worry too much about text formatting. A sample of the formatting follows.

Tables have also been extended from Markdown:

First Header	Second Header
Content Cell	Content Cell
Content Cell	Content Cell

Here’s an example of an image, which is included using Markdown:

Highlighting for code in Jekyll is done using Base16 or Rouge. This theme makes use of Rouge by default.

// count to ten
for (var i = 1; i <= 10; i++) {
    console.log(i);
}

// count to twenty
var j = 0;
while (j < 20) {
    j++;
    console.log(j);
}

Type on Strap uses KaTeX to display maths. Equations such as $S_n = a \times \frac{1-r^n}{1-r}$ can be displayed inline.

Alternatively, they can be shown on a new line:

$f(x) = \int \frac{2x^2+4x+6}{x-2}$

What's your title

Thu, 27 Nov 2014 00:00:00 +0000

This is an example of a post which includes a feature image that has a text, where you don’t want to redisplay the title. We cannot simply set the title to the empty string, as that would break pages that list this post, such as home and tags.

Markup: Syntax Highlighting

Fri, 08 Aug 2014 00:00:00 +0000

From Michael’s Rose Minimal Mistakes. Syntax highlighting is a feature that displays source code, in different colors and fonts according to the category of terms. This feature facilitates writing in a structured language such as a programming language or a markup language as both structures and syntax errors are visually distinct. Highlighting does not affect the meaning of the text itself; it is intended only for human readers.

GFM Code Blocks

GitHub Flavored Markdown fenced code blocks are supported. To modify styling and highlight colors edit /_sass/syntax.scss.

#container {
  float: left;
  margin: 0 -240px 0 0;
  width: 100%;
}

1
2
3
4
5
6
7
.highlight {
  margin: 0;
  padding: 1em;
  font-family: $monospace;
  font-size: $type-size-7;
  line-height: 1.8;
}

<nav class="pagination" role="navigation">
  {% if page.previous %}
    <a href="{{ site.url }}{{ page.previous.url }}" class="btn" title="{{ page.previous.title }}">Previous article</a>
  {% endif %}
  {% if page.next %}
    <a href="{{ site.url }}{{ page.next.url }}" class="btn" title="{{ page.next.title }}">Next article</a>
  {% endif %}
</nav><!-- /.pagination -->

module Jekyll
  class TagIndex < Page
    def initialize(site, base, dir, tag)
      @site = site
      @base = base
      @dir = dir
      @name = 'index.html'
      self.process(@name)
      self.read_yaml(File.join(base, '_layouts'), 'tag_index.html')
      self.data['tag'] = tag
      tag_title_prefix = site.config['tag_title_prefix'] || 'Tagged: '
      tag_title_suffix = site.config['tag_title_suffix'] || '&#8211;'
      self.data['title'] = "#{tag_title_prefix}#{tag}"
      self.data['description'] = "An archive of posts tagged #{tag}."
    end
  end
end

Code Blocks in Lists

Indentation matters. Be sure the indent of the code block aligns with the first non-space character after the list item marker (e.g., 1.). Usually this will mean indenting 3 spaces instead of 4.

Do step 1.

Now do this:

def print_hi(name)
  puts "Hi, #{name}"
end
print_hi('Tom')
#=> prints 'Hi, Tom' to STDOUT.

Now you can do this.

GitHub Gist Embed

An example of a Gist embed below.

Source

Generate a table of content

Thu, 12 Dec 2013 00:00:00 +0000

Test article, get the source on github.

Using Kramdown GFM

Using Kramdown GFM
- Basic Usage
Using toc.js

Basic Usage

You need to put this at the beginning of the page where you want the table of content to be displayed

* TOC
{:toc}

It will then render the markdown and html titles (lines that begins with # or using the <h1></h1> tages)

Using toc.js

Demo display of jekyll-table-of-contents by ghiculescu.

Customize with toc.js

toc.js stands for table of content, it is a js plugin that generates automatically a table of content of a post.

Use with this jekyll template

If you want to customize the theme it is up to you, you can add the toc.js file into the asset > js and add it into the page.html layout with:

<script src="/assets/js/toc.js" ></script>

Then you can use it as it is said on the repository.

Basic Usage

The script requires jQuery. First, reference toc.js in templates where you would like to add the table of content. Then, create an HTML element wherever you want your table of contents to appear:

<div id="toc"></div>

Then you put your post with titles and all like:

## Title
## Mid title 1
This is text on page one
## Mid title 2
This is text for page two
### Sub title 2.a
Some more text

Then at the end of your post, you call the .toc() function using:

<script type="text/javascript">
$(document).ready(function() {
    $('#toc').toc();
});
</script>

How it would look like

Blogging with title

Mon, 18 Nov 2013 00:00:00 +0000

I am a BIG title

This is a very tiny tiny post with less than 250 letters.

Search should be working even for complicated escape symbols

sed -i 's/\"hostname\"\:.*$/\"hostname\"\: \"'$IPADDR'\"\,/g' open-falcon/agent/config/cfg.json

Dong Liang's Blog

Sample post

Unsupervised Pre-training of Image Features on Non-Curated Data

Unsupervised Pre-training of Image Features on Non-Curated Data

Code && Paper

名词解释

问题定义

Related Work

DeeperClustering

idea

overview

Loss function

Invariant Information Clustering for Unsupervised Image Classification and Segmentation

Invariant Information Clustering for Unsupervised Image Classification and Segmentation

Code && Paper

问题定义

主要思想

相关工作

DeepCluster: Deep Clustering for Unsupervised Learning of Visual Features

UTS: Unsupervised Teacher-Student Model for Large-Scale Video Retrieval

IIC

overview

Loss function

为什么L1正则项产生稀疏的权重，L2正则项产生相对平滑的权重

L1和L2正则化的区别

Color Post

What a colorful post!

How does it work?

Feature images

Markdown and HTML

What's your title

Markup: Syntax Highlighting

GFM Code Blocks

Code Blocks in Lists

GitHub Gist Embed

Source

Generate a table of content

Using Kramdown GFM

Basic Usage

Using toc.js

Customize with toc.js

Use with this jekyll template

Basic Usage

How it would look like

Blogging with title

I am a BIG title