Code to reproduce my EMNLP 2021 paper, The Power of Scale for Parameter-Efficient Prompt Tuning. Built on T5X, a reimplementation of T5 in JAX, this code base allows for the training of prompts on Google Cloud TPUs. It also includes extended setting such as factored prompts and partial attention masking. It has been the foundation of our follow up reserach and should serve as a great starting point for your work.

A simple 2-bit Quantum Computer simulator in Python based on the Gate Quantum Computation model. It includes methods for combining qubits into a single representation, and for factoring that transformed vector back into a sequence of qubits. Representing \(n\) qubits as a single vector results in a vector of size \(2^{n}\), which is suggestive of the expoential memory requirements of simulating a quantum computer on a classical one. The math of the simulation also sheads light some of the weirder aspects of Quantum Mechanics. For example, entangled qubits results in an unfactorable vector. This in turn means that the qubits must be measured together, resulting in only the states where the qubits are equal (as is observed of entangled qubits in real life) being possible.

Core contributor to Mead-Baseline, a deep-learning toolkit for NLP that provids a uniform API to both TensorFlow and PyTorch backends, reusable implementations of complex neural architectures and layers, strong deep learning based models and good hyper-parameters, robust and correct evluation metrics, and performant batched implementations. Mead-Baseline uses Inversion of Control and an extensive plugin system to enable quick experiments by changing sections of model archetecutre with just a few lines of code. Model archetecture and training regime are specified in a configuration file to allow for repeatable research. Mead-Basline also enables easy access to a variety of contextual embeddings in the form of large pretrained language models like ELMo or BERT. Mead-Baseline also includes exporting utilities to convert models into either tensorflow serving servables or ONNX traced models which are ready to be deployed. Mead-Baseline is a one-stop shop that can take models all the way from prototyping to production.

My main work has been on the API design and implementation of the lower layers. I batched both the Conditional Random Field and the Beam Search yielding 6x speed ups. Before development on the framework basically stopped I created DyNet implementations of all the models in Mead-Baseline including large transformers. I introduced the ability to perform Constrained Decoding on the CRF where the rules of the transition scheme can be used to narrow down the number of possible transitions when doing Viterbi Decoding. Constrained Decoding gives gains across the board and should always be used.

A small library for parsing, converting, and manipulating span level NLP tasks (NER, slot-filling, etc) that are encoded as token level annotations like IOBES or BIO. A paper describing this library, and the need for standardized tools for span level NLP, was published at the second workshop on NLP Open Source Software.

An implementation of Text Rank in Python that reproduces the results from the original paper. I currently use it as a vehicle for personal research: investigating if deep learning based sentence similarities will yield better summaries. Results forthcoming.

A collection of various minimum edit distance algorithms as well as token based methods like Jaccard overlap. The algorithms are implemented in Cython and scale well. The library is able to compute minimum edit distances between entire Wikipedia pages in under asecond.

Implementations of Locality Sensitive Hashing. Supports using MinHash to approximate Jaccard similarity and Random Hyperplanes for a cosine based LSH. Also provides utilties to estimate optimal band size for some desired threshold of similarity. This library enables users to find similar items in very large corpora quickly.

Various path-finding algorithms via dynamic programming. Algorithms implemented include:

• Breath-First Search.
• Dijkstra's Algorithm.
• A* Search with a Euclidean Distance Heuristic.

Algoritms are agnostic to the implementation of the underlaying graph implementation. Both grid-based and adjaceny-based graphs are provided.

Path-finding with fuel-based constraints—effectivly a maximum path distance—and refill stations is also included. This is done in a two step procedure using a meta-graph. First minimum distance paths are found between all combinations of the staring position, the ending positions, and all refilling stations. Then, any path longer than the fuel capacity is removed. The result is a meta-graph of reachable points with edges weighted based on the length of the path. Then the same shortest-path algorithm is applied to this meta-graph to find a path between the start and end positions that travels through required refilling stations. The final path is then reconstructed by replacing the abstract path of Node a→Node b in the meta-graph with the actual path from a to be found in step one.

I have a collection of libraries hosted on PyPI, most enable narrow data science verticals but some are generic utilities.

• file-or-name

  A decorator that lets you mark a function parameter as a file. This will automatically open the file when the function is called. This lets you write easy to test function that assume pre-opened files but allows your users to pass in file paths as strings.

  It also includes the ability to automatically do shadow paging by pre-pending a write mode file with a s. This is done by proxying all the writes to a temporary file and then when you close the context manager the file was opened in the original file is replaced with the temporary file. This allows for atomic writes.

• word-vectors

  Utilities to read and write common word vector formats such as word2vec and glove. Also introduces a new format called Leader which is a binary format like word2vec but we prefix the word with its length. This allows for reading without having to iterate over characters like word2vec does at the cost of just 3 bytes of disk space per word. Our benchmarks show this format is faster for all common pre-trained embeddings. A paper describing the new embedding format, benchmarks done, and the library can be found here.

• inverted-index

  A small, extensible, in-memory inverted index to quickly added full text search to a script. It also includes a prompt_toolkit completer that will search the inverted index as you type. Selecting an option will replace the input with that value. This is useful for when there are a lot of options to choose from and you want to provide a fuzzy search when looking for the right one.

• get-mnist

  Lightweight utilities to download and load both the MNIST and Fashion MNIST datasets. The only dependency is NumPy.

• search-cli

  A GUI around surfraw that lets you open a search box from anywhere, it also lets you do a search where the query is whatever text is highlighted.

• prehashed

  A dictionary subclass that stores the sha1 hash of keys rather than the full key. This reduces storage space, for example if your keys are long documents you can store and retrieve stats about them without storing the whole document in memory.

• true-case

  An interface that gives the probability that a token should be capitalized. Probabilities were estimated from the CNN/DailyMail dataset.

• quick-knn

  An LSH implementation that supports Jaccard Similarity through MinHash and cosine similarity via Random Hyperplanes.

• string-distance

  Cython implementations of various string distance algorithms.

• delta-of-delta

  Pedagogical implementations of various time-stamp encoding algorithms.

Dependency parsing via Deep Learning models written in PyTorch. Supports training parsers via dynamic oracles using either the Arc Eager or the Arc Hybrid transition scheme or a Graph based parser using Biaffine Attention. Currently working on creating a shared interface between the two parser types with plans to release it as an open source library

A reimplementation of the paper “A Decomposable Attention Model for Natural Language Inference” in DyNet. I was able to reproduce the paper results via extensive hyper parameter tuning

Using a deep convolutional network steering angles are predicted from a front facing camera. The network is 20 layers deep and uses various network architecture modifications to decrease training times. These include residual connections and batch normalization.

Using a deep convolutional network, written in Tensorflow, multi-digit sequences (up to five digits) are classified. This is done using an end-to-end network that recognize the digits in one step rather than creating separate parts that localize the digits, segment them, and then classify each digit individually. This network was trained using the Stanford Street View House Numbers Dataset and achieves 93.89 percent accuracy on the test dataset. Read the full write up here

Implementation of the Transmission Control Protocol (RFC 793) completed for Data Communications and Computer Networking. This implementation is fully functional TCP implementation required to ensure both packet delivery as well as in-order delivery of packets. This implementation also supports multiple in flight packets with using the "Go Back N" strategy. This requires the Minet framework to function.

Particle interaction simulation written in C++. This simulation was written for parallel computation using the Message Passing Interface (MPI) and run on the Stampede super computer at Texas Advanced Computing Center. The simulation was carried out using the Ring Algorithm. Analysis of the program showed linear speed up as the number of processors was increased.

Distributed group-based file sharing system written in Java. This includes a multi-threaded group server that handles user authentication and distributes token to users. Multi-threaded file servers use the tokens to distribute files only to approved users. A GUI for clients to access files was also created. This Systems is secure as well. Cryptographic tools like symmetric and public key cryptography, Diffie Hellman key exchange and digital signatures are leverage to create secure authentication protocols that provide perfect forward security, repel man in the middle attacks, and provide plausible deniability. This system uses two-factor authentication via a Time based One Time Password (TOTP) scheme as outlined in RFC 6238.

A small example of how two parties can use some public information and a secret that each of them keep to derive a shared secret over public, un-trusted communication channels using Diffie-Hellman Key Exchange. Working from a shared prime number \(n\) and a primitve root modulo \(n\), \(g\), Alice and Bob can derive a shared key based on private information they each have, \(a\) and \(b\) (integers between \(1\) and \(n\)) respectively. Alice sends Bob \(g^a \;\mathrm{mod}\; n\) and Bob sends Alice \(g^b \;\mathrm{mod}\; n\). Bob then raises Alices results to the power of \(b\) while Alice raises Bob's to the power of \(a\). This results in both of them having a shared secret— \(g^{ab} \;\mathrm{mod}\; n\). An attacker, Eve, seeing only the public information can only calculate \(g^{a+b} \;\mathrm{mod}\; n\) without solving the descrete log problem.

Facial detection and recognition in video streams. Facial detection is done with dlib's CUDA accelerated histogram of ordered gradients frontal face detector to find faces and then dlib again to estimate facial landmarks. These landmarks are then transformed using an affine transformation in openCV to center the face in the image. Embeddings are then generated using a deep neural network written in torch from the OpenFace project at Carnegie Mellon University. These embeddings are can then be classified by various machine learning techniques. This project used a support vector machine from Scikit-learn to classify the faces in the image.

A compiler written in C to compile an Object-Oriented language called MINI-Java. Lexical analysis is done using code generated by Lex based on specified regular expressions. Syntax analysis is done using a Context Free Grammar and Yacc. This syntactic analysis produces an Abstract Syntax Tree that is then parsed in the Semantic analysis phase.

A simple program for breaking Vigenere Ciphers for homework 2 of CS 1653 at the University of Pittsburgh. This program works by tracking the distances between sets of repeating characters to try to estimate the length of the key. I didn't have time to implement frequency analysis to try to figure out what the key actually was. This program is written in C and includes a C implementaion of a Red-Black tree (used to track the offsets between repeating sequences of characters).

A web app that allows users to create a league, schedule matches, report and track results, and rank participants using the Elo ranking system. Once league rankings have been established integration with challonge allow users to run tournaments that are automatically seeded based on the Elo ranking of the players. This was created for CS 1520 in collaboration with Chris Price, David Robertson, and Troy Taylor. The app is written and deployed with Google App Engine. A live demo is found here.