# Turing Complete?

The point of stating that a mathematical model is Turing Complete is to reveal the capability of the model to perform any calculation, given a sufficient amount of resources (i.e. infinite), not to show whether a specific implementation of a model does have those resources. Non-Turing complete models would not be able to handle a specific set of calculations, even with enough resources, something that reveals a difference in the way the two models operate, even when they have limited resources. Of course, to prove this property, you have to do have to assume that the models are able to use an infinite amount of resources, but this property of a model is relevant even when resources are limited.

# LSTM Question

Sigmoids make sense for the gates, as they control how much of the signal is left into/out of the cell. Think of it as a percentage: how many percent of the input signal should I store in the cell (or put out of the cell). It doesn’t make sense to amplify a signal and write 110% of the current cell signal to the output. That’s not what the gates are for. Likewise, it doesn’t make sense for the input unit to say “the current input is 901% relevant for the memory cell, so please store it 9 times as strongly as usual”. If that were the case, the input/output weights would have caused the signal to be 900% stronger to begin with.

For the output activation, ReLU can of course be used. However you might easily run into numerical problems, given that gradients already need to be truncated oftentimes (and ReLU doesn’t dampen them the way sigmoids do). If I recall correctly Bengio’s lab has a paper somewhere where they use ReLU for RNNs and they said they had problems of this kind (I may be wrong though, and I’m unable to find the paper right now).

Also, one of the benefits of ReLUs is that they stop vanishing gradients. But LSTM was already designed not to suffer from that, to begin with. Given that you don’t have vanishing gradient problems, it comes down to the question whether relu is better than sigmoids on principle (because it can learn better functions) or because its main advantage is easier training. Of course, this is a simplified view, especially since LSTM was not originally designed to be “deep”. if you use many layers of lstms, you might still get vanishing gradients if you use sigmoids.

# UAI 上发表的 Deep Hybrid Models

Volodymyr Kuleshov, Stefano Ermon

Most methods in machine learning are described as either discriminative or generative. The former often attain higher predictive accuracy, while the latter are more strongly regularized and can deal with missing data.

Here, we propose a new framework to combine a broad class of discriminative and generative models, interpolating between the two extremes with a multiconditional likelihood objective.

Unlike previous approaches, we couple the two components through shared latent variables, and train using recent advances in variational inference.

Instantiating our framework with modern deep architectures gives rise to deep hybrid models, a highly flexible family that generalizes several existing models and is effective in the semi-supervised setting, where it results in improvements over the state of the art on the SVHN dataset.

# Thompson Sampling Is Asymptotically Optimal in General Environments

Jan Leike et al

We discuss a variant of Thompson sampling for nonparametric reinforcement learning in a countable classes of general stochastic environments.

These environments can be non-Markov, nonergodic, and partially observable. We show that Thompson sampling learns the environment class in the sense that (1) asymptotically its value converges to the optimal value in mean and (2) given a recoverability assumption regret is sublinear.

# Hello World

Welcome to Hexo! This is your very first post. Check documentation for more info. If you get any problems when using Hexo, you can find the answer in troubleshooting or you can ask me on GitHub.

## Quick Start

### Deploy to remote sites

Improve Your Python: Python Classes and Object Oriented Programming
The class is a fundamental building block in Python. It is the underpinning for not only many popular programs and libraries, but the Python standard library as well. Understanding what classes are, when to use them, and how they can be useful is essential, and the goal of this article. In the process, we’ll explore what the term Object-Oriented Programming means and how it ties together with Python classes.

Everything Is An Object…
What is the class keyword used for, exactly? Like its function-based cousin def, it concerns the definition of things. While def is used to define a function, class is used to define a class. And what is a class? Simply a logical grouping of data and functions (the latter of which are frequently referred to as “methods” when defined within a class).

What do we mean by “logical grouping”? Well, a class can contain any data we’d like it to, and can have any functions (methods) attached to it that we please. Rather than just throwing random things together under the name “class”, we try to create classes where there is a logical connection between things. Many times, classes are based on objects in the real world (like Customer or Product). Other times, classes are based on concepts in our system, like HTTPRequest or Owner.

Regardless, classes are a modeling technique; a way of thinking about programs. When you think about and implement your system in this way, you’re said to be performing Object-Oriented Programming. “Classes” and “objects” are words that are often used interchangeably, but they’re not really the same thing. Understanding what makes them different is the key to understanding what they are and how they work.

..So Everything Has A Class?
Classes can be thought of as blueprints for creating objects. When I define a Customer class using the class keyword, I haven’t actually created a customer. Instead, what I’ve created is a sort of instruction manual for constructing “customer” objects. Let’s look at the following example code:

class Customer(object):
“””A customer of ABC Bank with a checking account. Customers have the
following properties:

Attributes:
name: A string representing the customer's name.
balance: A float tracking the current balance of the customer's account.
"""

def __init__(self, name, balance=0.0):
"""Return a Customer object whose name is *name* and starting
balance is *balance*."""
self.name = name
self.balance = balance

def withdraw(self, amount):
"""Return the balance remaining after withdrawing *amount*
dollars."""
if amount > self.balance:
raise RuntimeError('Amount greater than available balance.')
self.balance -= amount
return self.balance

def deposit(self, amount):
"""Return the balance remaining after depositing *amount*
dollars."""
self.balance += amount
return self.balance


The class Customer(object) line does not create a new customer. That is, just because we’ve defined a Customer doesn’t mean we’ve created one; we’ve merely outlined the blueprint to create a Customer object. To do so, we call the class’s init method with the proper number of arguments (minus self, which we’ll get to in a moment).

So, to use the “blueprint” that we created by defining the class Customer (which is used to create Customer objects), we call the class name almost as if it were a function: jeff = Customer(‘Jeff Knupp’, 1000.0). This line simply says “use the Customer blueprint to create me a new object, which I’ll refer to as jeff.”

The jeff object, known as an instance, is the realized version of the Customer class. Before we called Customer(), no Customer object existed. We can, of course, create as many Customer objects as we’d like. There is still, however, only one Customer class, regardless of how many instances of the class we create.

self?
So what’s with that self parameter to all of the Customer methods? What is it? Why, it’s the instance, of course! Put another way, a method like withdraw defines the instructions for withdrawing money from some abstract customer’s account. Calling jeff.withdraw(100.0) puts those instructions to use on the jeff instance.

So when we say def withdraw(self, amount):, we’re saying, “here’s how you withdraw money from a Customer object (which we’ll call self) and a dollar figure (which we’ll call amount). self is the instance of the Customer that withdraw is being called on. That’s not me making analogies, either. jeff.withdraw(100.0) is just shorthand for Customer.withdraw(jeff, 100.0), which is perfectly valid (if not often seen) code.

init
self may make sense for other methods, but what about init? When we call init, we’re in the process of creating an object, so how can there already be a self? Python allows us to extend the self pattern to when objects are constructed as well, even though it doesn’t exactly fit. Just imagine that jeff = Customer(‘Jeff Knupp’, 1000.0) is the same as calling jeff = Customer(jeff, ‘Jeff Knupp’, 1000.0); the jeff that’s passed in is also made the result.

This is why when we call init, we initialize objects by saying things like self.name = name. Remember, since self is the instance, this is equivalent to saying jeff.name = name, which is the same as jeff.name = ‘Jeff Knupp. Similarly, self.balance = balance is the same as jeff.balance = 1000.0. After these two lines, we consider the Customer object “initialized” and ready for use.

Be careful what you init

After init has finished, the caller can rightly assume that the object is ready to use. That is, after jeff = Customer(‘Jeff Knupp’, 1000.0), we can start making deposit and withdraw calls on jeff; jeff is a fully-initialized object.

Imagine for a moment we had defined the Customer class slightly differently:

class Customer(object):
“””A customer of ABC Bank with a checking account. Customers have the
following properties:

Attributes:
name: A string representing the customer's name.
balance: A float tracking the current balance of the customer's account.
"""

def __init__(self, name):
"""Return a Customer object whose name is *name*."""
self.name = name

def set_balance(self, balance=0.0):
"""Set the customer's starting balance."""
self.balance = balance

def withdraw(self, amount):
"""Return the balance remaining after withdrawing *amount*
dollars."""
if amount > self.balance:
raise RuntimeError('Amount greater than available balance.')
self.balance -= amount
return self.balance

def deposit(self, amount):
"""Return the balance remaining after depositing *amount*
dollars."""
self.balance += amount
return self.balance


This may look like a reasonable alternative; we simply need to call set_balance before we begin using the instance. There’s no way, however, to communicate this to the caller. Even if we document it extensively, we can’t force the caller to call jeff.set_balance(1000.0) before calling jeff.withdraw(100.0). Since the jeff instance doesn’t even have a balance attribute until jeff.set_balance is called, this means that the object hasn’t been “fully” initialized.

The rule of thumb is, don’t introduce a new attribute outside of the init method, otherwise you’ve given the caller an object that isn’t fully initialized. There are exceptions, of course, but it’s a good principle to keep in mind. This is part of a larger concept of object consistency: there shouldn’t be any series of method calls that can result in the object entering a state that doesn’t make sense.

Invariants (like, “balance should always be a non-negative number”) should hold both when a method is entered and when it is exited. It should be impossible for an object to get into an invalid state just by calling its methods. It goes without saying, then, that an object should start in a valid state as well, which is why it’s important to initialize everything in the init method.

Instance Attributes and Methods
An function defined in a class is called a “method”. Methods have access to all the data contained on the instance of the object; they can access and modify anything previously set on self. Because they use self, they require an instance of the class in order to be used. For this reason, they’re often referred to as “instance methods”.

If there are “instance methods”, then surely there are other types of methods as well, right? Yes, there are, but these methods are a bit more esoteric. We’ll cover them briefly here, but feel free to research these topics in more depth.

Static Methods
Class attributes are attributes that are set at the class-level, as opposed to the instance-level. Normal attributes are introduced in the init method, but some attributes of a class hold for all instances in all cases. For example, consider the following definition of a Car object:

class Car(object):

wheels = 4

def __init__(self, make, model):
self.make = make
self.model = model


mustang = Car(‘Ford’, ‘Mustang’)
print mustang.wheels

print Car.wheels

# 4

A Car always has four wheels, regardless of the make or model. Instance methods can access these attributes in the same way they access regular attributes: through self (i.e. self.wheels).

There is a class of methods, though, called static methods, that don’t have access to self. Just like class attributes, they are methods that work without requiring an instance to be present. Since instances are always referenced through self, static methods have no self parameter.

The following would be a valid static method on the Car class:

class Car(object):

def make_car_sound():
print ‘VRooooommmm!’
No matter what kind of car we have, it always makes the same sound (or so I tell my ten month old daughter). To make it clear that this method should not receive the instance as the first parameter (i.e. self on “normal” methods), the @staticmethod decorator is used, turning our definition into:

class Car(object):

@staticmethod
def make_car_sound():
print ‘VRooooommmm!’
Class Methods
A variant of the static method is the class method. Instead of receiving the instance as the first parameter, it is passed the class. It, too, is defined using a decorator:

class Vehicle(object):

@classmethod
def is_motorcycle(cls):
return cls.wheels == 2
Class methods may not make much sense right now, but that’s because they’re used most often in connection with our next topic: inheritance.

Inheritance
While Object-oriented Programming is useful as a modeling tool, it truly gains power when the concept of inheritance is introduced. Inherticance is the process by which a “child” class derives the data and behavior of a “parent” class. An example will definitely help us here.

Imagine we run a car dealership. We sell all types of vehicles, from motorcycles to trucks. We set ourselves apart from the competition by our prices. Specifically, how we determine the price of a vehicle on our lot: $5,000 x number of wheels a vehicle has. We love buying back our vehicles as well. We offer a flat rate - 10% of the miles driven on the vehicle. For trucks, that rate is$10,000. For cars, $8,000. For motorcycles,$4,000.

If we wanted to create a sales system for our dealership using Object-oriented techniques, how would we do so? What would the objects be? We might have a Sale class, a Customer class, an Inventory class, and so forth, but we’d almost certainly have a Car, Truck, and Motorcycle class.

What would these classes look like? Using what we’ve learned, here’s a possible implementation of the Car class:

class Car(object):
“””A car for sale by Jeffco Car Dealership.

Attributes:
wheels: An integer representing the number of wheels the car has.
miles: The integral number of miles driven on the car.
make: The make of the car as a string.
model: The model of the car as a string.
year: The integral year the car was built.
sold_on: The date the vehicle was sold.
"""

def __init__(self, wheels, miles, make, model, year, sold_on):
"""Return a new Car object."""
self.wheels = wheels
self.miles = miles
self.make = make
self.model = model
self.year = year
self.sold_on = sold_on

def sale_price(self):
"""Return the sale price for this car as a float amount."""
if self.sold_on is not None:
return 5000.0 * self.wheels

def purchase_price(self):
"""Return the price for which we would pay to purchase the car."""
if self.sold_on is None:
return 0.0  # Not yet sold
return 8000 - (.10 * self.miles)

...


OK, that looks pretty reasonable. Of course, we would likely have a number of other methods on the class, but I’ve shown two of particular interest to us: sale_price and purchase_price. We’ll see why these are important in a bit.

Now that we’ve got the Car class, perhaps we should crate a Truck class? Let’s follow the same pattern we did for car:

class Truck(object):
“””A truck for sale by Jeffco Car Dealership.

Attributes:
wheels: An integer representing the number of wheels the truck has.
miles: The integral number of miles driven on the truck.
make: The make of the truck as a string.
model: The model of the truck as a string.
year: The integral year the truck was built.
sold_on: The date the vehicle was sold.
"""

def __init__(self, wheels, miles, make, model, year, sold_on):
"""Return a new Truck object."""
self.wheels = wheels
self.miles = miles
self.make = make
self.model = model
self.year = year
self.sold_on = sold_on

def sale_price(self):
"""Return the sale price for this truck as a float amount."""
if self.sold_on is not None:
return 5000.0 * self.wheels

def purchase_price(self):
"""Return the price for which we would pay to purchase the truck."""
if self.sold_on is None:
return 0.0  # Not yet sold
return 10000 - (.10 * self.miles)

...


Wow. That’s almost identical to the car class. One of the most important rules of programming (in general, not just when dealing with objects) is “DRY” or “Don’t Repeat Yourself. We’ve definitely repeated ourselves here. In fact, the Car and Truck classes differ only by a single character (aside from comments).

So what gives? Where did we go wrong? Our main problem is that we raced straight to the concrete: Cars and Trucks are real things, tangible objects that make intuitive sense as classes. However, they share so much data and functionality in common that it seems there must be an abstraction we can introduce here. Indeed there is: the notion of Vehicles.

Abstract Classes
A Vehicle is not a real-world object. Rather, it is a concept that some real-world objects (like cars, trucks, and motorcycles) embody. We would like to use the fact that each of these objects can be considered a vehicle to remove repeated code. We can do that by creating a Vehicle class:

class Vehicle(object):
“””A vehicle for sale by Jeffco Car Dealership.

Attributes:
wheels: An integer representing the number of wheels the vehicle has.
miles: The integral number of miles driven on the vehicle.
make: The make of the vehicle as a string.
model: The model of the vehicle as a string.
year: The integral year the vehicle was built.
sold_on: The date the vehicle was sold.
"""

base_sale_price = 0

def __init__(self, wheels, miles, make, model, year, sold_on):
"""Return a new Vehicle object."""
self.wheels = wheels
self.miles = miles
self.make = make
self.model = model
self.year = year
self.sold_on = sold_on

def sale_price(self):
"""Return the sale price for this vehicle as a float amount."""
if self.sold_on is not None:
return 5000.0 * self.wheels

def purchase_price(self):
"""Return the price for which we would pay to purchase the vehicle."""
if self.sold_on is None:
return 0.0  # Not yet sold
return self.base_sale_price - (.10 * self.miles)


Now we can make the Car and Truck class inherit from the Vehicle class by replacing object in the line class Car(object). The class in parenthesis is the class that is inherited from (object essentially means “no inheritance”. We’ll discuss exactly why we write that in a bit).

We can now define Car and Truck in a very straightforward way:

class Car(Vehicle):

def __init__(self, wheels, miles, make, model, year, sold_on):
"""Return a new Car object."""
self.wheels = wheels
self.miles = miles
self.make = make
self.model = model
self.year = year
self.sold_on = sold_on
self.base_sale_price = 8000


class Truck(Vehicle):

def __init__(self, wheels, miles, make, model, year, sold_on):
"""Return a new Truck object."""
self.wheels = wheels
self.miles = miles
self.make = make
self.model = model
self.year = year
self.sold_on = sold_on
self.base_sale_price = 10000


This works, but has a few problems. First, we’re still repeating a lot of code. We’d ultimately like to get rid of all repetition. Second, and more problematically, we’ve introduced the Vehicle class, but should we really allow people to create Vehicle objects (as opposed to Cars or Trucks)? A Vehicle is just a concept, not a real thing, so what does it mean to say the following:

v = Vehicle(4, 0, ‘Honda’, ‘Accord’, 2014, None)
print v.purchase_price()
A Vehicle doesn’t have a base_sale_price, only the individual child classes like Car and Truck do. The issue is that Vehicle should really be an Abstract Base Class. Abstract Base Classes are classes that are only meant to be inherited from; you can’t create instance of an ABC. That means that, if Vehicle is an ABC, the following is illegal:

v = Vehicle(4, 0, ‘Honda’, ‘Accord’, 2014, None)
It makes sense to disallow this, as we never meant for vehicles to be used directly. We just wanted to use it to abstract away some common data and behavior. So how do we make a class an ABC? Simple! The abc module contains a metaclass called ABCMeta (metaclasses are a bit outside the scope of this article). Setting a class’s metaclass to ABCMeta and making one of its methods virtual makes it an ABC. A virtual method is one that the ABC says must exist in child classes, but doesn’t necessarily actually implement. For example, the Vehicle class may be defined as follows:

from abc import ABCMeta, abstractmethod

class Vehicle(object):
“””A vehicle for sale by Jeffco Car Dealership.

Attributes:
wheels: An integer representing the number of wheels the vehicle has.
miles: The integral number of miles driven on the vehicle.
make: The make of the vehicle as a string.
model: The model of the vehicle as a string.
year: The integral year the vehicle was built.
sold_on: The date the vehicle was sold.
"""

__metaclass__ = ABCMeta

base_sale_price = 0

def sale_price(self):
"""Return the sale price for this vehicle as a float amount."""
if self.sold_on is not None:
return 5000.0 * self.wheels

def purchase_price(self):
"""Return the price for which we would pay to purchase the vehicle."""
if self.sold_on is None:
return 0.0  # Not yet sold
return self.base_sale_price - (.10 * self.miles)

@abstractmethod
def vehicle_type():
""""Return a string representing the type of vehicle this is."""
pass


Now, since vehicle_type is an abstractmethod, we can’t directly create an instance of Vehicle. As long as Car and Truck inherit from Vehicle and define vehicle_type, we can instantiate those classes just fine.

Returning to the repetition in our Car and Truck classes, let see if we can’t remove that by hoisting up common functionality to the base class, Vehicle:

from abc import ABCMeta, abstractmethod
class Vehicle(object):
“””A vehicle for sale by Jeffco Car Dealership.

Attributes:
wheels: An integer representing the number of wheels the vehicle has.
miles: The integral number of miles driven on the vehicle.
make: The make of the vehicle as a string.
model: The model of the vehicle as a string.
year: The integral year the vehicle was built.
sold_on: The date the vehicle was sold.
"""

__metaclass__ = ABCMeta

base_sale_price = 0
wheels = 0

def __init__(self, miles, make, model, year, sold_on):
self.miles = miles
self.make = make
self.model = model
self.year = year
self.sold_on = sold_on

def sale_price(self):
"""Return the sale price for this vehicle as a float amount."""
if self.sold_on is not None:
return 5000.0 * self.wheels

def purchase_price(self):
"""Return the price for which we would pay to purchase the vehicle."""
if self.sold_on is None:
return 0.0  # Not yet sold
return self.base_sale_price - (.10 * self.miles)

@abstractmethod
def vehicle_type(self):
""""Return a string representing the type of vehicle this is."""
pass


Now the Car and Truck classes become:

class Car(Vehicle):
“””A car for sale by Jeffco Car Dealership.”””

base_sale_price = 8000
wheels = 4

def vehicle_type(self):
""""Return a string representing the type of vehicle this is."""
return 'car'


class Truck(Vehicle):
“””A truck for sale by Jeffco Car Dealership.”””

base_sale_price = 10000
wheels = 4

def vehicle_type(self):
""""Return a string representing the type of vehicle this is."""
return 'truck'


This fits perfectly with our intuition: as far as our system is concerned, the only difference between a car and truck is the base sale price. Defining a Motorcycle class, then, is similarly simple:

class Motorcycle(Vehicle):
“””A motorcycle for sale by Jeffco Car Dealership.”””

base_sale_price = 4000
wheels = 2

def vehicle_type(self):
""""Return a string representing the type of vehicle this is."""
return 'motorcycle'


Inheritance and the LSP
Even though it seems like we used inheritance to get rid of duplication, what we were really doing was simply providing the proper level of abstraction. And abstraction is the key to understanding inheritance. We’ve seen how one side-effect of using inheritance is that we reduce duplicated code, but what about from the caller’s perspective. How does using inheritance change that code?

Quite a bit, it turns out. Imagine we have two classes, Dog and Person, and we want to write a function that takes either type of object and prints out whether or not the instance in question can speak (a dog can’t, a person can). We might write code like the following:

def can_speak(animal):
if isinstance(animal, Person):
return True
elif isinstance(animal, Dog):
return False
else:
raise RuntimeError(‘Unknown animal!’)
That works when we only have two types of animals, but what if we have twenty, or two hundred? That if…elif chain is going to get quite long.

The key insight here is that can_speak shouldn’t care what type of animal it’s dealing with, the animal class itself should tell us if it can speak. By introducing a common base class, Animal, that defines can_speak, we relieve the function of it’s type-checking burden. Now, as long as it knows it was an Animal that was passed in, determining if it can speak is trivial:

def can_speak(animal):
return animal.can_speak()
This works because Person and Dog (and whatever other classes we crate to derive from Animal) follow the Liskov Substitution Principle. This states that we should be able to use a child class (like Person or Dog) wherever a parent class (Animal) is expected an everything will work fine. This sounds simple, but it is the basis for a powerful concept we’ll discuss in a future article: interfaces.

Summary
Hopefully, you’ve learned a lot about what Python classes are, why they’re useful, and how to use them. The topic of classes and Object-oriented Programming are insanely deep. Indeed, they reach to the core of computer science. This article is not meant to be an exhaustive study of classes, nor should it be your only reference. There are literally thousands of explanations of OOP and classes available online, so if you didn’t find this one suitable, certainly a bit of searching will reveal one better suited to you.

As always, corrections and arguments are welcome in the comments. Just try to keep it civil.

Lastly, it’s not too late to see me speak at the upcoming Wharton Web Conference at UPenn! Check the site for info and tickets.

# 1 为何学习生成式建模？

• 训练生成式模型和从生成式模型进行采样是我们表示和操纵高维概率分布的能力的特别好的检验。高维概率分布在很多的应用数学和工程领域都是举足轻重的研究对象。
• 生成式模型可以被以多种方式用在强化学习中。强化学习算法常常会被分类成两类：基于模型和免模型的，而基于模型的算法就是包含生成式模型的。时间序列数据的生成式模型可以被用来模型可能的未来。这样的模型可以被用来进行规划和多种方式的强化学习任务。用于规划的生成式模型可以学到在给定世界当前的状态和假设智能体会采取的行动作为输入时，关于世界的未来状态的条件分布。而智能体可以用不同的潜在行动来查询这个模型，选择模型预测为最可能从此行动得到想要的状态的行动。最近出来的这种模型例子是，Finn 等人的研究工作，而对把这个模型用于规划的例子就是 Finn 和 Levine 的研究工作。另一种生成式模型可以被用在强化学习上的方式是在一个想象的环境中进行学习，其中错误的行动不会给智能体带来真实的损失。生成式模型还可以通过追踪不同状态被访问的频率或者不同的行动被尝试的频率来指导探索，特别是 GANs，可以用在逆强化学习中，在第 5.6 节我们会讲其与强化学习的关联。
• 生成式模型可以用 missing 数据进行训练，并能够给出输入数据的 missing 部分。而 missing 数据的特别有趣的例子就是半监督学习（semi-supervised learning），其中很多（或者几乎所有）训练样本的标签都是丢失的。现代深度学习算法一般是要求有特别多的带标签样本才能够泛化得很好。半监督学习是降低样本标签的策略。这个学习算法可以通过研究大量无标签样本提升自己的泛化性能，而这些无标签样本是很容易获得的。生成式模型尤其是 GANs，能够很好地执行半监督学习算法。这个会在第 5.4 节介绍。

• 生成式模型尤其是 GANs，让机器学习能够更好地利用多模态输出。对很多任务，单个输入可能会对应不同的正确答案，这些答案中的每一个都是可以接受的。某些传统训练机器学习模型的方式，如最小化目标输出和模型预测输出的均方误差，并不能够训练出可以产生多个不同的正确答案的模型。这种场景的一个例子就是预测视频的下一帧，参见图 3。
• 最后，很多任务本质上都是需要某个分布中的采样的样本的。

• 单个图片超分辨率：这个任务的目标是以低分辨率图像为输入，合成高分辨率的相同内容的图片。生成式建模的作用就是让模型能够为图像加入原本就该属于输入的信息。有很多对应于低分辨图像的超分辨图像。这个模型应该选择一副采样自可能的图像的概率分布的图像。选择一幅是所有可能图像的平均图像可能会得到一个太过模糊的结果。参见图 4。

• 一些创作艺术的任务。两个近期项目表明生成式模型，尤其是 GANs，可以用于创建交互式程序来帮助用户创作对应于他们想象的粗线条场景的更加真实的图像。参见图 5 和图 6.

• 图像到图像的转换应用可以转换航摄照片成地图或者将线条转换为图像。有很多困难但是很有用的创造性的应用。参见图 7。

# 在基于能量的模型中使用前期推断近似反向传播

Early Inference in Energy-Based Models Approximates Back-Propagation(Arxiv)

Montreal Institute for Learning Algorithms, University of Montreal

## 摘要

minicolumn，人脑中最小的组件；A cortical minicolumn is a vertical column through the cortical layers of the brain, comprising perhaps 80–120 neurons, except in the primate primary visual cortex (V1), where there are typically more than twice the number.

## 1. 引言

leaky integrator: A kind of deliberately imperfect integration used in numerical work is called leaky integration. The name derives from the analogous situation of electrical circuits, where the voltage on a capacitor is the integral of the current: in real life, some of the current leaks away. An equation to model leaky integration is

## 相关工作，贡献和未来工作

and Seung, 2006; Rezende and Gerstner, 2014) 依赖于 强化学习观点来估计全局目标函数的梯度（通过关联在每个神经元随机变化和整个目标函数的改变）。尽管这些原理是简单的，还不清楚这会扩展到非常大的网络，因为随神经元个数变化导致估计的方差线性增长。所以尝试寻找其他的途径，我们希望本文提出的基础和能量观点下的变分推断可以形成有用的更加有效的无监督学习的原理，用于那些和 STDP 一致的深层网络。

Welling (2014); Bengio et al. (2015b)。通过最小化预测标准如 $J_{STDP}$ 我们推测，模型参数移动的方向让模型和 $Q(h,x)$ 更为一致，这也就能够最大化变分 EM 在数据似然 $P(x)$ 的上界。这个想法就是我们改变推断过程，使得它能够更快地抵达最终状态，这对应于可以很好地拟合观测 $x$ 的 $h$ 的设定。

## Acknowledgments

The authors would like to thank Benjamin Scellier, Asja Fischer, Thomas Mesnard, Saizheng Zhang, Yuhuai Wu, Dong-Hyun Lee, Jyri Kivinen, Jorg Bornschein, Roland Memisevic and Tim Lillicrap for feedback and discussions, as well as NSERC, CIFAR, Samsung and Canada Research Chairs for funding.

## References

1. Alain, G. and Bengio, Y. (2013). What regularized autoencoders learn from the data generating distribution. In ICLR’2013. also arXiv report 1211.4246.
2. Andrieu, C., de Freitas, N., Doucet, A., and Jordan, M. (2003). An introduction to MCMC for machine learning. Machine Learning, 50, 5–43.
3. Bengio, Y., Mesnard, T., Fischer, A., Zhang, S., and Wu, Y. (2015a). An objective function for stdp. arXiv:1509.05936.
4. Bengio, Y., Lee, D.-H., Bornschein, J., and Lin, Z. (2015b). Towards biologically plausible deep learning. arXiv:1502.04156.
5. Berkes, P., Orban, G., Lengyel, M., and Fiser, J. (2011).
Spontaneous cortical activity reveals hallmarks of an optimal internal model of the environment. Science, 331, 83––87.
6. Fiete, I. R. and Seung, H. S. (2006). Gradient learning in spiking neural networks by dynamic perturbations of conductances. Physical Review Letters, 97(4).
7. Friston, K. J. and Stephan, K. E. (2007). Free-energy and the brain. Synthese, 159, 417––458.
8. Hinton, G. E. (2007). How to do backpropagation in a brain. Invited talk at the NIPS’2007 Deep Learning Workshop.
9. Hinton, G. E. and Sejnowski, T. J. (1986). Learning and relearning in Boltzmann machines. In D. E. Rumelhart and J. L. McClelland, editors, Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 1: Foundations, pages 282–317. MIT Press, Cambridge, MA.
10. Kingma, D. P. and Welling, M. (2014). Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representations (ICLR).
11. Markram, H., Gerstner, W., and Sj¨ostr¨om, P. (2012). Spiketiming-dependent plasticity: A comprehensive overview. Frontiers in synaptic plasticity, 4(2).
12. Neal, R. and Hinton, G. (1999). A view of the EM algorithm that justifies incremental, sparse, and other variants. In M. I. Jordan, editor, Learning in Graphical Models. MIT Press, Cambridge, MA.
13. Rezende, D. J. and Gerstner, W. (2014). Stochastic variational learning in recurrent spiking networks. Frontiers in Computational Neuroscience, 8(38).
14. Vincent, P. (2011). A connection between score matching and denoising autoencoders. Neural Computation, 23(7).
15. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., and Manzagol, P.-A. (2010). Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J. Machine Learning Res., 11.
16. Williams, R. J. (1992). Simple statistical gradient following algorithms connectionist reinforcement learning.
Machine Learning, 8, 229–256.

# Q-学习

$$\hat{Q}_{\mathrm{opt}}(s,a) \leftarrow \hat{Q}_{\mathrm{opt}}(s,a) - \eta [\color{red}{\underbrace{\hat{Q}_{\mathrm{opt}}(s,a)}_\mathrm{prediction}} - \color{green}{\underbrace{(r + \gamma\hat{V}_{\mathrm{opt}}(s’))}_\mathrm{target}}]$$

$\color{red}{问题就是}$ 无法泛化到未见过的状态/行动上

# 函数近似

$\color{red}{关键想法：线性回归模型}$

$\hat{Q}_{\mathrm{opt}}(s,a;\mathbf{w}) = \color{blue}{\mathbf{w} \cdot \phi(s,a)}$

$\color{brown}{例子：过火山问题中的特征}$

• $\phi_1(s,a) = \mathbf{1}[a=W]$
• $\phi_2(s,a) = \mathbf{1}[a=E]$
• $\phi_7(s,a) = \mathbf{1}[s = (5,*)]$
• $\phi_8(s,a) = \mathbf{1}[s = (*,6)]$
• 函数近似通过用一个权重向量和一个特征向量来参数化 $\hat{Q}_{\mathrm{opt}}$ 解决了上面的问题，这个其实和线性回归中一样.

• 特征可以看成是状态-行动 $(s,a)$ 对的属性，可以作为在状态 $s$ 采取行动 $a$ 的质量的衡量.
• 不过这样做的后果就是所有有着类似的特征的状态会有相似的 Q-值. 例如，假设 $\phi$ 包含特征 $\mathbf{1}[s=(*,4)]$. 如果我们在状态 $(1,4)$，采取行动 $\mathbf{E}$，并获得高奖励，那么采用函数近似的 Q-学习会将这个正的信号传播到在第 4 列的所有位置上采取任何行动.
• 在这个例子中，我们在行动上定义特征（为了刻画向东移动通常是好的选择）而在状态上定义特征（为刻画第 6 列是最好要避开的事实，而第 5 行是好的移动位置）

$\color{blue}{算法：采用函数近似的 Q-学习}$

$\mathbf{w} \leftarrow \eta[\color{red}{\underbrace{\hat{Q}_{\mathrm{opt}}(s,a;\mathbf{w})}_\mathrm{prediction}} - \color{green}{\underbrace{(r + \gamma\hat{V}_{\mathrm{opt}}(s’))}_\mathrm{target}}] \color{blue}{\phi(s,a)}$

$$\min_\mathbf{w} \sum_{(s,a,r,s’)}\Big(\color{red}{\underbrace{\hat{Q}_{\mathrm{opt}}(s,a;\mathbf{w})}_\mathrm{prediction}} - \color{green}{\underbrace{(r + \gamma\hat{V}_{\mathrm{opt}}(s’))}_\mathrm{target}}\Big)^2$$

• 现在我们就把线性回归搞成了一个算法. 这里，应用 RL 的随机梯度方法就有效了
• 我们刚刚写下了最小平方目标函数，接着计算有关 $\mathbf{w}$ 的梯度而不是之前的 $\hat{Q}_{\mathrm{opt}}$. 链式法则处理剩下的事情了.

# 深度强化学习

• 最后 4 帧（图像）=> 3-层网络 => 控制杆
• $\epsilon$-贪婪，使用 1 百万回放空间训练 1 千万帧的图像

• 最近，由于深度学习的成功，又出现了对强化学习的兴趣. 如果某个人在一个模拟器上执行强化学习方法，那么就可以产生大量的数据，利于神经网络发挥作用.

• 近期成功的故事来自 DeepMind，他们成功地训练出一个神经网络来表示玩 Atari 游戏的 $\hat{Q}_{\mathrm{opt}}$. 这里令人印象深刻的部分是不需要先验知识：神经网络简单地以原始图像作为输入并输出控制杆的选择.

# 了解未知世界

• Epsilon-greedy：平衡探索和开发
• 函数近似：可以泛化到未见的状态

# Summary so far

• 在线的设定: 在真实世界中学习和行动 learn and take actions in the real world!