Alle afleveringen van Machine Learning Street Talk (MLST) – pagina 6 van 6

Capsule Networks and Education Targets

29 september 2020 · 1 uur 24 min

Afspelen

In today's episode, Dr. Keith Duggar, Alex Stenlake and Dr. Tim Scarfe chat about the education chapter in Kenneth Stanley's "Greatness cannot be planned" book, and we relate it to our Algoshambes conversation a few weeks ago. We debate whether objectives in education are a good thing and whether they cause perverse incentives and stifle creativity and innovation. Next up we dissect capsule networks from the top down! We finish off talking about fast algorithms and quantum computing. Lees meer

00:00:00 Introduction

00:01:13 Greatness cannot be planned / education

00:12:03 Perverse incentives

00:19:25 Treasure hunting

00:30:28 Capsule Networks

00:46:08 Capsules As Compositional Networks

00:52:45 Capsule Routing

00:57:10 Loss and Warps

01:09:55 Fast Algorithms and Quantum Computing

Programming Languages, Software Engineering and Machine Learning

25 september 2020 · 1 uur 24 min

Afspelen

This week Dr. Tim Scarfe, Dr. Keith Duggar, Yannic "Lightspeed" Kilcher have a conversation with Microsoft Senior Software Engineer Sachin Kundu. We speak about programming languages including which our favourites are and functional programming vs OOP. Next we speak about software engineering and the intersection of software engineering and machine learning. We also talk about applications of ML and finally what makes an exceptional software engineer and tech lead. Sachin is an expert in this field so we hope you enjoy the conversation! Lees meer

Spoiler alert, how many of you have read the Mythical Man-Month by Frederick P. Brooks?!

00:00:00 Introduction

00:06:37 Programming Languages

00:53:41 Applications of ML

01:55:59 What makes an exceptional SE and tech lead

01:22:08 Outro

Computation, Bayesian Model Selection, Interactive Articles

23 september 2020 · 1 uur 14 min

Afspelen

This week Dr. Keith Duggar, Alex Stenlake and Dr. Tim Scarfe discuss the theory of computation, intelligence, Bayesian model selection, the intelligence explosion and the the phenomenon of "interactive articles". Lees meer

00:00:00 Intro

00:01:27 Kernels and context-free grammars

00:06:04 Theory of computation

00:18:41 Intelligence

00:22:03 Bayesian model selection

00:44:05 AI-IQ Measure / Intelligence explosion

00:52:09 Interactive articles

01:12:32 Outro

Kernels!

18 september 2020 · 1 uur 37 min

Afspelen

Today Yannic Lightspeed Kilcher and I spoke with Alex Stenlake about Kernel Methods. What is a kernel? Do you remember those weird kernel things which everyone obsessed about before deep learning? What about Representer theorem and reproducible kernel hilbert spaces? SVMs and kernel ridge regression? Remember them?! Hope you enjoy the conversation! Lees meer

00:00:00 Tim Intro

00:01:35 Yannic clever insight from this discussion

00:03:25 Street talk and Alex intro

00:05:06 How kernels are taught

00:09:20 Computational tractability

00:10:32 Maths

00:11:50 What is a kernel?

00:19:39 Kernel latent expansion

00:23:57 Overfitting

00:24:50 Hilbert spaces

00:30:20 Compare to DL

00:31:18 Back to hilbert spaces

00:45:19 Computational tractability 2

00:52:23 Curse of dimensionality

00:55:01 RBF: infinite taylor series

00:57:20 Margin/SVM

01:00:07 KRR/dual

01:03:26 Complexity compute kernels vs deep learning

01:05:03 Good for small problems? vs deep learning)

01:07:50 Whats special about the RBF kernel

01:11:06 Another DL comparison

01:14:01 Representer theorem

01:20:05 Relation to back prop

01:25:10 Connection with NLP/transformers

01:27:31 Where else kernels good

01:34:34 Deep learning vs dual kernel methods

01:33:29 Thoughts on AI

01:34:35 Outro

Explainability, Reasoning, Priors and GPT-3

16 september 2020 · 1 uur 26 min

Afspelen

This week Dr. Tim Scarfe and Dr. Keith Duggar discuss Explainability, Reasoning, Priors and GPT-3. We check out Christoph Molnar's book on intepretability, talk about priors vs experience in NNs, whether NNs are reasoning and also cover articles by Gary Marcus and Walid Saba critiquing deep learning. We finish with a brief discussion of Chollet's ARC challenge and intelligence paper. Lees meer

00:00:00 Intro

00:01:17 Explainability and Christoph Molnars book on Intepretability

00:26:45 Explainability - Feature visualisation

00:33:28 Architecture / CPPNs

00:36:10 Invariance and data parsimony, priors and experience, manifolds

00:42:04 What NNs learn / logical view of modern AI (Walid Saba article)

00:47:10 Core knowledge

00:55:33 Priors vs experience

00:59:44 Mathematical reasoning

01:01:56 Gary Marcus on GPT-3

01:09:14 Can NNs reason at all?

01:18:05 Chollet intelligence paper/ARC challenge

SWaV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (Mathilde Caron)

14 september 2020 · 1 uur 28 min

Afspelen

This week Dr. Tim Scarfe, Yannic Lightspeed Kicher, Sayak Paul and Ayush Takur interview Mathilde Caron from Facebook Research (FAIR). Lees meer

We discuss Mathilde's paper which she wrote with her collaborators "SWaV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments" @ https://arxiv.org/pdf/2006.09882.pdf

This paper is the latest unsupervised contrastive visual representations algorithm and has a new data augmentation strategy and also a new online clustering strategy.

Note; Other authors; Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, Armand Joulin

Sayak Paul - @RisingSayak / https://www.linkedin.com/in/sayak-paul/

Ayush Thakur - @ayushthakur0

/ https://www.linkedin.com/in/ayush-thakur-731914149/

The article they wrote;

https://app.wandb.ai/authors/swav-tf/reports/Unsupervised-Visual-Representation-Learning-with-SwAV--VmlldzoyMjg3Mzg

00:00:00 Yannic probability challenge (CAN YOU SOLVE IT?)

00:01:29 Intro topic (Tim)

00:08:18 Yannic take

00:09:33 Intro show and guests

00:11:29 SWaV elevator pitch

00:17:31 Clustering approach in general

00:21:17 Sayak and Ayush's article on SWaV

00:23:49 Optional transport problem / Sinkhorn-Knopp algorithm

00:31:43 Is clustering a natural approach for this?

00:44:19 Image augmentations

00:46:20 Priors vs experience (data)

00:48:32 Life at FAIR

00:52:33 Progress of image augmentation

00:56:10 When things do not go to plan with research

01:01:04 Question on architecture

01:01:43 SWaV Results

01:06:26 Reproducing Matilde's code

01:14:51 Do we need the whole dataset to set clustering loss

01:16:40 Self-supervised learning and transfer learning

01:23:25 Link to attention mechanism)

01:24:41 Sayak final thought why unsupervised better

01:25:56 Outro

Abstract;

"Unsupervised image representations have significantly reduced the gap with supervised pretraining, notably with the recent achievements of contrastive learning methods. These contrastive methods typically work online an…

UK Algoshambles, Neuralink, GPT-3 and Intelligence

7 september 2020 · 1 uur 35 min

Afspelen

This week Dr. Tim Scarfe, Dr. Keith Duggar and Yannic "Lightspeed" Kilcher respond to the "Algoshambles" exam fiasco in the UK where the government were forced to step in to standardise the grades which were grossly inflated by the schools. The schools and teachers are all paid on metrics related to the grades received by students, what could possibly go wrong?! The result is that we end up with grades which have lost all their value and students are coached for the exams and don't actually learn the subject. We also cover the second Francois Chollet interview on the Lex Fridman podcast. We cover GPT-3, Neuralink, and discussion of intelligence. Lees meer

00:00:00 Algoshambles

00:45:40 Lex Fridman/Chollet: Intro

00:55:21 Lex Fridman/Chollet: Neuralink

01:06:28 Lex Fridman/Chollet: GPT-3

01:23:43 Lex Fridman/Chollet: Intelligence discussion

Sayak Paul

17 juli 2020 · 1 uur 36 min

Afspelen

This week we spoke with Sayak Paul, who is extremely active in the machine learning community. We discussed the AI landscape in India, unsupervised representation learning, data augmentation and contrastive learning, explainability, abstract scene representations and finally pruning and the recent super positions paper. I really enjoyed this conversation and I hope you folks do too! Lees meer

00:00:00 Intro to Sayak

00:17:50 AI landscape in India

00:24:20 Unsupervised representation learning

00:26:11 DATA AUGMENTATION/Contrastive learning

00:59:20 EXPLAINABILITY

01:12:10 ABSTRACT SCENE REPRESENTATIONS

01:14:50 PRUNING and super position paper

Robert Lange on NN Pruning and Collective Intelligence

8 juli 2020 · 1 uur 46 min

Afspelen

We speak with Robert Lange! Lees meer

Robert is a PhD student at the Technical University Berlin. His research combines Deep Multi-Agent Reinforcement Learning and Cognitive Science to study the learning dynamics of large collectives. He has a brilliant blog where he distils and explains cutting edge ML research. We spoke about his story, economics, multi-agent RL, intelligence and AGI, and his recent article summarising the state of the art in neural network pruning.

Robert's article on pruning in NNs https://roberttlange.github.io/posts/2020/06/lottery-ticket-hypothesis/

00:00:00 Intro

00:04:17 Show start and intro to Robert

00:11:39 Economics background

00:27:20 Intrinsic motivation

00:33:22 Intelligence/consciousness

00:48:16 Lottery ticket/pruning article discussion

01:43:21 Robert's advice for younger self and state of deep learning

Robert's LinkedIn: https://www.linkedin.com/in/robert-tjarko-lange-19539a12a/

@RobertTLange

#machinelearning #deeplearning

WelcomeAIOverlords (Zak Jost)

30 juni 2020 · 1 uur 58 min

Afspelen

We welcome Zak Jost from the WelcomeAIOverlords channel. Zak is an ML research scientist at Amazon. He has a great blog at http://blog.zakjost.com and also a Discord channel at https://discord.gg/xh2chKX Lees meer

WelcomeAIOverlords: https://www.youtube.com/channel/UCxw9_WYmLqlj5PyXu2AWU_g

00:00:00 INTRO START

00:01:07 MAIN SHOW START

00:01:59 ZAK'S STORY

00:05:06 YOUTUBE DISCUSSION

00:24:12 UNDERSTANDING PAPERS

00:29:53 CONTRASTIVE LEARNING INTRO

00:33:00 BRING YOUR OWN LATENT PAPER

01:03:13 GRAPHS IN ML AND KNOWLEDGE GRAPHS

01:21:36 GRAPH USE CASES - FRAUD

01:30:15 KNOWLEDGE GRAPHS

01:34:22 GRAPHS IN ML

01:38:53 AUTOMATED ML

01:57:32 OUTRO

Facebook Research - Unsupervised Translation of Programming Languages

24 juni 2020 · 1 uur 3 min

Afspelen

In this episode of Machine Learning Street Talk Dr. Tim Scarfe, Yannic Kilcher and Connor Shorten spoke with Marie-Anne Lachaux, Baptiste Roziere and Dr. Guillaume Lample from Facebook Research (FAIR) in Paris. They recently released the paper "Unsupervised Translation of Programming Languages" which was an exciting new approach to learned translation of programming languages (learned transcoder) using an unsupervised encoder trained on individual monolingual corpora i.e. no parallel language data needed. The trick they used what that there is significant token overlap when using word-piece embeddings. It was incredible to talk with this talented group of researchers and I hope you enjoy the conversation too. Lees meer

Yannic's video on this got watched over 120K times! Check it out too https://www.youtube.com/watch?v=xTzFJIknh7E

Paper https://arxiv.org/abs/2006.03511;

Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanussot, Guillaume Lample

Abstract;

"A transcompiler, also known as source-to-source translator, is a system that converts source code from a high-level programming language (such as C++ or Python) to another. Transcompilers are primarily used for interoperability, and to port codebases written in an obsolete or deprecated language (e.g. COBOL, Python 2) to a modern one. They typically rely on handcrafted rewrite rules, applied to the source code abstract syntax tree. Unfortunately, the resulting translations often lack readability, fail to respect the target language conventions, and require manual modifications in order to work properly. The overall translation process is timeconsuming and requires expertise in both the source and target languages, making code-translation projects expensive. Although neural models significantly outperform their rule-based counterparts in the context of natural language translation, their applications to transcompilation have been limited due to the scarcity of parallel data in this domain. In this paper, we propose to levera…

Francois Chollet - On the Measure of Intelligence

19 juni 2020 · 2 uur 34 min

Afspelen

We cover Francois Chollet's recent paper. Lees meer

Abstract; To make deliberate progress towards more intelligent and more human-like artificial systems, we need to be following an appropriate feedback signal: we need to be able to define and evaluate intelligence in a way that enables comparisons between two systems, as well as comparisons with humans. Over the past hundred years, there has been an abundance of attempts to define and measure intelligence, across both the fields of psychology and AI. We summarize and critically assess these definitions and evaluation approaches, while making apparent the two historical conceptions of intelligence that have implicitly guided them. We note that in practice, the contemporary AI community still gravitates towards benchmarking intelligence by comparing the skill exhibited by AIs and humans at specific tasks such as board games and video games. We argue that solely measuring skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience: unlimited priors or unlimited training data allow experimenters to "buy" arbitrary levels of skills for a system, in a way that masks the system's own generalization power. We then articulate a new formal definition of intelligence based on Algorithmic Information Theory, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience. Using this definition, we propose a set of guidelines for what a general AI benchmark should look like. Finally, we present a benchmark closely following these guidelines, the Abstraction and Reasoning Corpus (ARC), built upon an explicit set of priors designed to be as close as possible to innate human priors. We argue that ARC can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans.

OpenAI GPT-3: Language Models are Few-Shot Learners

7 juni 2020 · 1 uur 52 min

Afspelen

In this episode of Machine Learning Street Talk, Tim Scarfe, Yannic Kilcher and Connor Shorten discuss their takeaways from OpenAI’s GPT-3 language model. With the help of Microsoft’s ZeRO-2 / DeepSpeed optimiser, OpenAI trained an 175 BILLION parameter autoregressive language model. The paper demonstrates how self-supervised language modelling at this scale can perform many downstream tasks without fine-tuning. Lees meer

00:00:00 Intro

00:00:54 ZeRO1+2 (model + Data parallelism) (Connor)

00:03:17 Recent history of NLP (Tim)

00:06:04 Yannic "Light-speed" Kilcher's brief overview of GPT-3

00:14:25 Reviewing Yannic's YT comments on his GPT-3 video (Tim)

00:20:26 Main show intro

00:23:03 Is GPT-3 reasoning?

00:28:15 Architecture discussion and autoregressive (GPT*) vs denoising autoencoder (BERT)

00:36:18 Utility of GPT-3 in industry

00:43:03 Can GPT-3 do math? (reasoning/system 1/system 2)

00:51:03 Generalisation

00:56:48 Esoterics of language models

00:58:46 Architectural trade-offs

01:07:37 Memorization machines and intepretability

01:17:16 Nearest neighbour probes / watermarks

01:20:03 YouTube comments on GPT-3 video

01:21:50 GPT-3 news article generation issue

01:27:36 Sampling data for language models / bias / fairness / politics

01:51:12 Outro

These paradigms of task adaptation are divided into zero, one, and few shot learning. Zero-shot learning is a very extreme case where we expect a language model to perform a task such as sentiment classification or extractive question answering, without any additional supervision. One and Few-shot learning provide some examples to the model. However, GPT-3s definition of this diverges a bit from the conventional literature. GPT-3 provides one and few-shot examples in the form of “In-Context Learning”. Instead of fine-tuning the model on a few examples, the model has to use the input to infer the downstream task. For example, the GPT-3 transformer has an input sequence of 2048 tokens, so demonstrations of a task …

Jordan Edwards: ML Engineering and DevOps on AzureML

3 juni 2020 · 1 uur 13 min

Afspelen

This week we had a super insightful conversation with Jordan Edwards, Principal Program Manager for the AzureML team! Jordan is on the coalface of turning machine learning software engineering into a reality for some of Microsoft's largest customers. Lees meer

ML DevOps is all about increasing the velocity of- and orchastrating the non-interactive phase of- software deployments for ML. We cover ML DevOps and Microsoft Azure ML. We discuss model governance, testing, intepretability, tooling. We cover the age-old discussion of the dichotomy between science and engineering and how you can bridge the gap with ML DevOps. We cover Jordan's maturity model for ML DevOps.

We also cover off some of the exciting ML announcments from the recent Microsoft Build conference i.e. FairLearn, IntepretML, SEAL, WhiteNoise, OpenAI code generation, OpenAI GPT-3.

00:00:04 Introduction to ML DevOps and Microsoft Build ML Announcements

00:10:29 Main show kick-off

00:11:06 Jordan's story

00:14:36 Typical ML DevOps workflow

00:17:38 Tim's articulation of ML DevOps

00:19:31 Intepretability / Fairness

00:24:31 Testing / Robustness

00:28:10 Using GANs to generate testing data

00:30:26 Gratuitous DL?

00:33:46 Challenges of making an ML DevOps framework / IaaS

00:38:48 Cultural battles in ML DevOps

00:43:04 Maturity Model for Ml DevOps

00:49:19 ML: High interest credit card of technical debt paper

00:50:19 ML Engineering at Microsoft

01:01:20 ML Flow

01:03:05 Company-wide governance

01:08:15 What's coming next

01:12:10 Jordan's hillarious piece of advice for his younger self

Super happy with how this turned out, this is not one to miss folks!

#deeplearning #machinelearning #devops #mldevops

One Shot and Metric Learning - Quadruplet Loss (Machine Learning Dojo)

2 juni 2020 · 2 uur 28 min

Afspelen

*Note this is an episode from Tim's Machine Learning Dojo YouTube channel. Lees meer

Join Eric Craeymeersch on a wonderful discussion all about ML engineering, computer vision, siamese networks, contrastive loss, one shot learning and metric learning.

00:00:00 Introduction

00:11:47 ML Engineering Discussion

00:35:59 Intro to the main topic

00:42:13 Siamese Networks

00:48:36 Mining strategies

00:51:15 Contrastive Loss

00:57:44 Trip loss paper

01:09:35 Quad loss paper

01:25:49 Eric's Quadloss Medium Article

02:17:32 Metric learning reality check

02:21:06 Engineering discussion II

02:26:22 Outro

In our second paper review call, Tess Ferrandez covered off the FaceNet paper from Google which was a one-shot siamese network with the so called triplet loss. It was an interesting change of direction for NN architecture i.e. using a contrastive loss instead of having a fixed number of output classes. Contrastive architectures have been taking over the ML landscape recently i.e. SimCLR, MOCO, BERT.

Eric wrote an article about this at the time: https://medium.com/@crimy/one-shot-learning-siamese-networks-and-triplet-loss-with-keras-2885ed022352

He then discovered there was a new approach to one shot learning in vision using a quadruplet loss and metric learning. Eric wrote a new article and several experiments on this @ https://medium.com/@crimy/beyond-triplet-loss-one-shot-learning-experiments-with-quadruplet-loss-16671ed51290?source=friends_link&sk=bf41673664ad8a52e322380f2a456e8b

Paper details:

Beyond triplet loss: a deep quadruplet network for person re-identification

https://arxiv.org/abs/1704.01719 (Chen at al '17)

"Person re-identification (ReID) is an important task in wide area video surveillance which focuses on identifying people across different cameras. Recently, deep learning networks with a triplet loss become a common framework for person ReID. However, the triplet loss pays main attentions on obtaining correct orders on the training set. It still suf…

Harri Valpola: System 2 AI and Planning in Model-Based Reinforcement Learning

25 mei 2020 · 1 uur 38 min

Afspelen

In this episode of Machine Learning Street Talk, Tim Scarfe, Yannic Kilcher and Connor Shorten interviewed Harri Valpola, CEO and Founder of Curious AI. We continued our discussion of System 1 and System 2 thinking in Deep Learning, as well as miscellaneous topics around Model-based Reinforcement Learning. Dr. Valpola describes some of the challenges of modelling industrial control processes such as water sewage filters and paper mills with the use of model-based RL. Dr. Valpola and his collaborators recently published “Regularizing Trajectory Optimization with Denoising Autoencoders” that addresses some of the concerns of planning algorithms that exploit inaccuracies in their world models! Lees meer

00:00:00 Intro to Harri and Curious AI System1/System 2

00:04:50 Background on model-based RL challenges from Tim

00:06:26 Other interesting research papers on model-based RL from Connor

00:08:36 Intro to Curious AI recent NeurIPS paper on model-based RL and denoising autoencoders from Yannic

00:21:00 Main show kick off, system 1/2

00:31:50 Where does the simulator come from?

00:33:59 Evolutionary priors

00:37:17 Consciousness

00:40:37 How does one build a company like Curious AI?

00:46:42 Deep Q Networks

00:49:04 Planning and Model based RL

00:53:04 Learning good representations

00:55:55 Typical problem Curious AI might solve in industry

01:00:56 Exploration

01:08:00 Their paper - regularizing trajectory optimization with denoising

01:13:47 What is Epistemic uncertainty

01:16:44 How would Curious develop these models

01:18:00 Explainability and simulations

01:22:33 How system 2 works in humans

01:26:11 Planning

01:27:04 Advice for starting an AI company

01:31:31 Real world implementation of planning models

01:33:49 Publishing research and openness

We really hope you enjoy this episode, please subscribe!

Regularizing Trajectory Optimization with Denoising Autoencoders: https://papers.nips.cc/paper/8552-regularizing-trajectory-optimization-with-denoising-…

ICLR 2020: Yoshua Bengio and the Nature of Consciousness

22 mei 2020 · 2 uur 34 min

Afspelen

In this episode of Machine Learning Street Talk, Tim Scarfe, Connor Shorten and Yannic Kilcher react to Yoshua Bengio’s ICLR 2020 Keynote “Deep Learning Priors Associated with Conscious Processing”. Bengio takes on many future directions for research in Deep Learning such as the role of attention in consciousness, sparse factor graphs and causality, and the study of systematic generalization. Bengio also presents big ideas in Intelligence that border on the line of philosophy and practical machine learning. This includes ideas such as consciousness in machines and System 1 and System 2 thinking, as described in Daniel Kahneman’s book “Thinking Fast and Slow”. Similar to Yann LeCun’s half of the 2020 ICLR keynote, this talk takes on many challenging ideas and hopefully this video helps you get a better understanding of some of them! Thanks for watching! Lees meer

Please Subscribe for more videos!

Paper Links:

Link to Talk: https://iclr.cc/virtual_2020/speaker_7.html

The Consciousness Prior: https://arxiv.org/abs/1709.08568

Thinking Fast and Slow: https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555

Systematic Generalization: https://arxiv.org/abs/1811.12889

CLOSURE: Assessing Systematic Generalization of CLEVR Models: https://arxiv.org/abs/1912.05783

Neural Module Networks: https://arxiv.org/abs/1511.02799

Experience Grounds Language: https://arxiv.org/pdf/2004.10151.pdf

Benchmarking Graph Neural Networks: https://arxiv.org/pdf/2003.00982.pdf

On the Measure of Intelligence: https://arxiv.org/abs/1911.01547

Please check out our individual channels as well!

Machine Learning Dojo with Tim Scarfe: https://www.youtube.com/channel/UCXvHuBMbgJw67i5vrMBBobA

Yannic Kilcher: https://www.youtube.com/channel/UCZHmQk67mSJgfCCTn7xBfe

Henry AI Labs: https://www.youtube.com/channel/UCHB9VepY6kYvZjj0Bgxnpbw

00:00:00 Tim and Yannics takes

00:01:37 Intro to Bengio

00:03:13 System 2, language and Chomsky

00:05:58 Cristof Koch on conciousness

00:07:25 Franco…

ICLR 2020: Yann LeCun and Energy-Based Models

20 mei 2020 · 2 uur 12 min

Afspelen

This week Connor Shorten, Yannic Kilcher and Tim Scarfe reacted to Yann LeCun's keynote speech at this year's ICLR conference which just passed. ICLR is the number two ML conference and was completely open this year, with all the sessions publicly accessible via the internet. Yann spent most of his talk speaking about self-supervised learning, Energy-based models (EBMs) and manifold learning. Don't worry if you hadn't heard of EBMs before, neither had we! Lees meer

Thanks for watching! Please Subscribe!

Paper Links:

ICLR 2020 Keynote Talk: https://iclr.cc/virtual_2020/speaker_7.html

A Tutorial on Energy-Based Learning: http://yann.lecun.com/exdb/publis/pdf/lecun-06.pdf

Concept Learning with Energy-Based Models (Yannic's Explanation): https://www.youtube.com/watch?v=Cs_j-oNwGgg

Concept Learning with Energy-Based Models (Paper): https://arxiv.org/pdf/1811.02486.pdf

Concept Learning with Energy-Based Models (OpenAI Blog Post): https://openai.com/blog/learning-concepts-with-energy-functions/

#deeplearning #machinelearning #iclr #iclr2020 #yannlecun

The Lottery Ticket Hypothesis with Jonathan Frankle

20 mei 2020 · 1 uur 27 min

Afspelen

In this episode of Machine Learning Street Talk, we chat with Jonathan Frankle, author of The Lottery Ticket Hypothesis. Frankle has continued researching Sparse Neural Networks, Pruning, and Lottery Tickets leading to some really exciting follow-on papers! This chat discusses some of these papers such as Linear Mode Connectivity, Comparing and Rewinding and Fine-tuning in Neural Network Pruning, and more (full list of papers linked below). We also chat about how Jonathan got into Deep Learning research, his Information Diet, and work on developing Technology Policy for Artificial Intelligence! Lees meer

This was a really fun chat, I hope you enjoy listening to it and learn something from it!

Thanks for watching and please subscribe!

Huge thanks to everyone on r/MachineLearning who asked questions!

Paper Links discussed in the chat:

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks: https://arxiv.org/abs/1803.03635

Linear Mode Connectivity and the Lottery Ticket Hypothesis: https://arxiv.org/abs/1912.05671

Dissecting Pruned Neural Networks: https://arxiv.org/abs/1907.00262

Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs: https://arxiv.org/abs/2003.00152

What is the State of Neural Network Pruning? https://arxiv.org/abs/2003.03033

The Early Phase of Neural Network Training: https://arxiv.org/abs/2002.10365

Comparing Rewinding and Fine-tuning in Neural Network Pruning: https://arxiv.org/abs/2003.02389

(Also Mentioned)

Block-Sparse GPU Kernels: https://openai.com/blog/block-sparse-gpu-kernels/

Balanced Sparsity for Efficient DNN Inference on GPU: https://arxiv.org/pdf/1811.00206.pdf

Playing the Lottery with Rewards and Multiple Languages: Lottery Tickets in RL and NLP: https://arxiv.org/pdf/1906.02768.pdf

r/MachineLearning question list: https://www.reddit.com/r/MachineLearning/comments/g9jqe0/d_lottery_ticket_hypothesis_ask_the_author_a/ (edited)

#machinelearning #deeplearning

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

19 mei 2020 · 1 uur 40 min

Afspelen

In this episode of Machine Learning Street Talk, Tim Scarfe, Yannic Kilcher and Connor Shorten chat about Large-scale Transfer Learning in Natural Language Processing. The Text-to-Text Transfer Transformer (T5) model from Google AI does an exhaustive survey of what’s important for Transfer Learning in NLP and what’s not. In this conversation, we go through the key takeaways of the paper, text-to-text input/output format, architecture choice, dataset size and composition, fine-tuning strategy, and how to best use more computation. Lees meer

Beginning with these topics, we diverge into exciting ideas such as embodied cognition, meta-learning, and the measure of intelligence. We are still beginning our podcast journey and really appreciate any feedback from our listeners. Is the chat too technical? Do you prefer group discussions, interviewing experts, or chats between the three of us? Thanks for watching and if you haven’t already, Please Subscribe!

Paper Links discussed in the chat:

Text-to-Text Transfer Transformer: https://arxiv.org/abs/1910.10683

Experience Grounds Language (relevant to divergent discussion about embodied cognition): https://arxiv.org/pdf/2004.10151.pdf

On the Measure of Intelligence: https://arxiv.org/abs/1911.01547

Train Large, Then Compress: https://arxiv.org/pdf/2002.11794.pdf

Scaling Laws for Neural Language Models: https://arxiv.org/pdf/2001.08361.pdf

The Illustrated Transformer: http://jalammar.github.io/illustrated...

ELECTRA: https://arxiv.org/pdf/2003.10555.pdf

Transformer-XL: https://arxiv.org/pdf/1901.02860.pdf

Reformer: The Efficient Transformer: https://openreview.net/pdf?id=rkgNKkHtvB

The Evolved Transformer: https://arxiv.org/pdf/1901.11117.pdf

DistilBERT: https://arxiv.org/pdf/1910.01108.pdf

How to generate text (HIGHLY RECOMMEND): https://huggingface.co/blog/how-to-ge...

Tokenizers: https://blog.floydhub.com/tokenization-nlp/

CURL: Contrastive Unsupervised Representations for Reinforcement Learning

2 mei 2020 · 1 uur 15 min

Afspelen

According to Yann Le Cun, the next big thing in machine learning is unsupervised learning. Self-supervision has changed the entire game in the last few years in deep learning, first transforming the language world with word2vec and BERT -- but now it's turning computer vision upside down. Lees meer

This week Yannic, Connor and I spoke with one of the authors, Aravind Srinivas who recently co-led the hot-off-the-press CURL: Contrastive Unsupervised Representations for Reinforcement Learning alongside Michael (Misha) Laskin. CURL has had an incredible reception in the ML community in the last month or so. Remember the Deep Mind paper which solved the Atari games using the raw pixels? Aravind's approach uses contrastive unsupervised learning to featurise the pixels before applying RL. CURL is the first image-based algorithm to nearly match the sample-efficiency and performance of methods that use state-based features! This is a huge step forwards in being able to apply RL in the real world.

We explore RL and self-supervision for computer vision in detail and find out about how Aravind got into machine learning.

Original YouTube Video: https://youtu.be/1MprzvYNpY8

Paper:

CURL: Contrastive Unsupervised Representations for Reinforcement Learning

Aravind Srinivas, Michael Laskin, Pieter Abbeel

https://arxiv.org/pdf/2004.04136.pdf

Yannic's analysis video: https://www.youtube.com/watch?v=hg2Q_O5b9w4

#machinelearning #reinforcementlearning #curl #timscarfe #yannickilcher #connorshorten

Music credit; https://soundcloud.com/errxrmusic/in-my-mind

Exploring Open-Ended Algorithms: POET

24 april 2020 · 1 uur 13 min

Afspelen

Three YouTubers; Tim Scarfe - Machine Learning Dojo (https://www.youtube.com/channel/UCXvHuBMbgJw67i5vrMBBobA), Connor Shorten - Henry AI Labs (https://www.youtube.com/channel/UCHB9VepY6kYvZjj0Bgxnpbw) and Yannic Kilcher (https://www.youtube.com/channel/UCZHmQk67mSJgfCCTn7xBfew). We made a new YouTube channel called Machine Learning Street Talk. Every week we will talk about the latest and greatest in AI. Subscribe now! Lees meer

Special guests this week; Dr. Mathew Salvaris (https://www.linkedin.com/in/drmathewsalvaris/), Eric Craeymeersch (https://www.linkedin.com/in/ericcraeymeersch/), Dr. Keith Duggar (https://www.linkedin.com/in/dr-keith-duggar/), Dmitri Soshnikov (https://www.linkedin.com/in/shwars/)

We discuss the new concept of an open-ended, or "AI-Generating" algorithm. Open-endedness is a class of algorithms which generate problems and solutions to increasingly complex and diverse tasks. These algorithms create their own curriculum of learning. Complex tasks become tractable because they are now the final stepping stone in a lineage of progressions. In many respects, it's better to trust the machine to develop the learning curriculum, because the best curriculum might be counter-intuitive. These algorithms can generate a radiating tree of evolving challenges and solutions just like natural evolution. Evolution has produced an eternity of diversity and complexity and even produced human intelligence as a side-effect! Could AI-generating algorithms be the next big thing in machine learning?

Wang, Rui, et al. "Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions." arXiv preprint arXiv:2003.08536 (2020). https://arxiv.org/abs/2003.08536

Wang, Rui, et al. "Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions." arXiv preprint arXiv:1901.01753 (2019). https://arxiv.org/abs/1901.01753

Watch Yannic’s video on POET: https:/…