Tony Cricelli

About Me

Hi, I'm Tony Cricelli, an Aerospace Engineer by education, an entrepreneur when needed, and a systems administrator by choice. Currently, I reside in the beautiful state of California.

I have a passion for systems administration, particularly in setting up networked Unix-based systems. My journey began with MS-DOS 2.11, followed by experiences with IBM Mainframes VM/CMS and Amdahl Mainframes using z/VM. During my tenure with the US Navy, I supported flight testing and learned to use XOS on Xerox mainframes, which was quite limited as the only interface available was cards or tapes. I later transitioned to VAX VMS on Digital Equipment Corporation computers, which ignited my love for networking systems and their administration. After several years, I expanded my expertise to SUN OS, managing Sun Sparc 10s and 20s, as well as Silicon Graphics Irix OS. Irix OS was particularly enjoyable due to its GL and GLX GUI capabilities. My first personal computer was a monochrome AT&T Unix machine that came equipped with C and FORTRAN compilers. Although it wasn't as fast as the SGI and Sun computers, it was uniquely mine. When I learned about Linus Torvalds, I decided to explore Linux.

Downloading 23 images and writing them to 3.5-inch floppy disks was quite the challenge, especially with Linux Kernel .96 at the time. I would FTP into poorsas.helsinki.fi to retrieve the necessary files. Back then, if Linux "crashed," it would wipe the drive clean, forcing you to restart the entire process of downloading disk images. I eagerly followed the development of GNU utilities, witnessing the evolution of a completely free operating system and X-Windows-based window managers, which I found fascinating. I often tell people that I "grew up" on Linux.

While Linux has always been my preference, I also utilized Windows 3.11, Windows 95, Windows 98, Windows NT, Windows 2000, Windows XP, Windows Vista, Windows 7, 10, and 11. I never felt completely comfortable with the limitations of the "Windows" interface. Recently, I've been using Windows 10 and 11 more frequently due to the built-in WSL (Windows Subsystem for Linux), which allows me to RDP into a Windows machine and access a Bash shell for my work.

I started my first business with a Professor Kolar called ICC, we were one of the earliest computer consultants in the Monterey I was working on my Master's degree at the time and after a few years we both lost interest. Also at the time, Buliten board systems were on decline and this new thing called the Internet was all the Rage. teamed up with a couple of friends and started an Internet company called mbay.net. Before we even had the first customer, I teamed up with a couple of other friends and started a competing Internet Company called Red Shift. It was actually a DBA of an existing Corporation. That was in the early 1990s. I still own the corporation and it is still doing business as RedShift.com. It now a leasing company. We lease our own data centers and mountain top tower locations. I am really not involved too much, I have people that manage the business so I am freed up to be a network engineer and systems administrator.

Education

Master of Science in Aerospace Engineering, Naval Postgraduate School, Monterey, California.

Bachelor of Science in Aerospace Engineering, Pennsylvania State University, State College, Pennsylvania.

Bachelor of Arts in General Arts and Sciences, Pennsylvania State University, State College, Pennsylvania.

Projects

2025 - Present: Terraforming an Open OnDemand Cluster on Azure

I discovered a GitHub repository that facilitated the deployment of an Open OnDemand cluster using Azure/Az-hop. Learning the intricacies of deployment took time, particularly in understanding the available VM sizes and GPU options in the selected region. Many assume that requested cloud resources will always be available. After weeks of effort, I managed to successfully access NVIDIA A100 and H100 GPUs, with Microsoft Support confirming my access to NVIDIA T4 GPUs. The cluster is now operational and awaiting utilization by researchers at U.C. Berkeley.

2024: Open OnDemand Cluster on Google Cloud Platform

This project marked my initial foray into Open OnDemand. Building the cluster on GCP initially appeared daunting. However, I discovered a Marketplace solution where a company had completed the Terraform work, allowing me to simply "purchase" the product. After deployment, I filled out the web form specifying my desired cluster size, disk size, RAM, GPU, etc., which proved straightforward after multiple iterations of building and dismantling the cluster. This cluster was designed to access GPUs, but I encountered challenges with the NVIDIA H100. Although the region had access to the H100s, they were perpetually unavailable when I attempted to initiate a VM with the H100 attached. I did have access to smaller NVIDIA GPUs, but the requirement was for 80GB of RAM with the H100. Thus, this Open OnDemand cluster remains fully operational, albeit primarily needed if the H100s become accessible.

2023: Deploying a Kubernetes Cluster on GCP

Instructors and researchers at U.C. Berkeley sought to implement Jupyter notebooks for educational and research purposes. They initially believed this would be straightforward, given that all students possessed computers and were provided with links to download the "notebook." However, they soon realized that essential libraries such as Python, R, and Julia were not pre-installed, making installation cumbersome. Instructors found themselves dedicating most of their office hours to troubleshooting and debugging environment setups instead of focusing on course material. Consequently, Jupyterhub was introduced, offering a web-based platform for users to run their own "notebooks" on a cluster of computers with prebuilt environments. These environments included all necessary libraries for Jupyter notebooks. I created custom Dockerfiles tailored for the specific course and pushed the images to hub.docker.com. Utilizing HELM and kubectl, I installed the course within a custom namespace on the cluster. To optimize costs, the cluster was designed to activate servers as student logins increased. While this approach seemed promising, it often fell short in practice. When 150 students attempted to log in simultaneously, GCP would initiate the required number of servers, leading to delays of up to 15 minutes for class sessions lasting only 45 minutes. I deemed this a failure, despite everything functioning as intended. Ultimately, customer satisfaction is paramount, and students expressed frustration with the wait times. A viable solution would be to maintain enough servers to accommodate student logins without delays, but this proved financially unfeasible, prompting U.C. Berkeley to request the cluster's dismantling.

2022 - Present: Deploying a Kubernetes Cluster on Bare Metal

This project is particularly gratifying as it grants me complete control over the environment. I began with 13 Dell R630 servers, ensuring firmware updates on each server and configuring RAID1 OS disks for maximum uptime. Each server is equipped with quad 10Gig interfaces for network connectivity, allowing me to design the cluster layout to my specifications.

I initiated the process by installing the latest version of Ubuntu with a minimal install. I utilized a couple of 24-port 10Gig switches in the rack to create four VLANs. One VLAN is dedicated to storage, connecting to my 150 TB NAS, while the second VLAN facilitates internode communication. The third VLAN serves as my management Kubernetes network, with all nodes communicating over the private network. I deployed RFC 1918 addresses across all VLANs.

Next, I designated one node as my primary node, connecting it to a public network. I created SSH keys and installed the public key in the authorized_keys file for all minion nodes. Additionally, I modified the /etc/ssh/sshd_config file to permit root login.

Subsequently, I installed Ansible, which is crucial for ensuring uniformity among nodes, as workloads are distributed throughout the cluster. I developed playbooks to install necessary tools and perform apt upgrades. These playbooks simplify the addition of new nodes in the future, eliminating the need for me to reference notes on required installations—just run the playbooks!

The playbooks I created facilitate Ubuntu software installations and updates, including one for Docker and another for Kubernetes.

I then exported my storage to the relevant IPs on the storage network and mounted the /home directory on all nodes.

Next, I initialized Kubernetes using kubeadm, employing the command generated by kubeadm to join all minion nodes via the token and management network.

With the cluster operational, I conducted a test nginx deployment, which successfully deployed three replicas. After dismantling the deployment, I installed Nginx and configured it as a reverse proxy, allowing for unlimited deployments using a single public IP while all workloads run on the private cluster.

With all components in place, I was prepared to deploy a custom Jupyterhub. A professor from U.C. Berkeley provided a list of required Python and R libraries for the course. I began constructing a custom Dockerfile and building it locally on my Mac. Once I achieved a functional image, I pushed it to hub.docker.com. After uploading, I utilized helm and kubectl to install Jupyterhub. Kubernetes assigned a private Cluster IP, which I integrated into the Nginx config file, providing it with a proper DNS name. I then employed certbot to obtain a free SSL certificate.

Once installed, I restarted Nginx, and like magic, Jupyterhub began responding to requests. This project has been running smoothly for two years, supporting up to 500 users simultaneously. Each user is allocated up to 6GB of RAM and four cores.

Community

I dedicate a significant amount of time to volunteering at local non-profits. Here is a brief list:

Hobbies

I enjoy playing golf; I'm not particularly skilled, but I find it enjoyable to spend time chatting with friends away from the outside world.

I also take pleasure in learning and experimenting with new technologies. In 2024, I began working with Raspberry Pi 5s equipped with 5G hats, which introduced me to ARM processors. Currently, I am building autonomous drones using the RPi5 and Pixhawk 6c flight controller. Planning missions, sending them to the drone, and watching it take off, capture photos and videos, and return home is incredibly rewarding.

While I used to ride horses and race motorcycles, I’ve stepped back from those activities. Though I enjoyed both immensely, I recognize that as I age, I may not recover as quickly from accidents.

Now, my only motorsport activities involve working on my cars and participating in occasional track days at Laguna Seca Raceway. Recently, I opted to have my car's ECU reprogrammed to create a more linear air/fuel ratio across all RPMs, from 0 to 9000 RPM. The programming was straightforward once I figured out how to remove the right and left ECUs. After completing the initial start-up procedures, the car performed well on the street. However, during a visit to Laguna Seca, I experienced a brief power loss at high RPMs, around 8500 RPM. After consulting with the ECU programmers and reviewing the code, they discovered I had received the race car mapping instead of the street car mapping. The race car has larger MAFs, allowing for more air and fuel. Although my fuel injectors were working, there wasn’t enough air to burn it. Faced with a choice between reprogramming the ECUs again or upgrading the MAFs, I opted for the latter. The MAFs were more accessible, so I decided to keep the race car mapping. I had already installed a high-flow TUBI exhaust and high-flow catalytic converters. After ordering and installing the new MAFs, I encountered another issue: they didn’t connect to the air intake boxes, as the airbox was too small. I ordered a custom carbon fiber airbox top that fit perfectly with the MAFs. I am thrilled with the results; my Ferrari is now as enjoyable at low speeds as my Fiat 500!

Contact

You can reach me at: montereytony@gmail.com

Please refrain from searching for me on GitHub; I keep my projects private :)