Ladvien's Lab

This article walks through building a working health data pipeline that automatically syncs your Apple Health data to a PostgreSQL database using FastAPI. No fluff, just the code that actually works.

The Problem: Health Data are Trapped

I've been wearing an Apple Watch for years, collecting thousands of data points about my heart rate, sleep, workouts, and more. But here's the thing—all that valuable data just sits there in the Health app, basically useless for any real analysis.

As a data engineer, this drives me nuts. I want to run SQL queries against my health data. I want to ...

If you use a Cloudflare Tunnel (instead of a public IP), traditional Let’s Encrypt HTTP challenges will fail. This guide walks you through replacing a Cloudflare Origin Certificate with a Let’s Encrypt certificate using Certbot with the DNS-01 challenge and the Cloudflare API.

Prerequisites

Ensure you have:

• A domain (e.g., self-sensored-api.lolzlab.com )
• DNS managed by Cloudflare
• A working Raspberry Pi with Nginx installed
• sudo or root access
• A Cloudflare account with API token permissions

Backup and edit your Nginx config:

sudo cp /etc/nginx/sites-available/reverse-proxy.conf ~/reverse-proxy.conf.backup
sudo nano /etc/nginx/sites-available/reverse-proxy ...

This guide walks through setting up a secure HTTPS connection between your local Raspberry Pi reverse proxy and Cloudflare using a Cloudflare Origin Certificate . This enables Full SSL mode , improving security end-to-end.

0. Prerequisites

• Cloudflare account with a configured domain (e.g. <your-host-name>.com )
• Cloudflared tunnel already running on your gateway device
• Nginx reverse proxy running on a separate Raspberry Pi (e.g. 192.168.1.102 )
• Cloudflare SSL/TLS mode set to Full (not Flexible)

1. The Architecture

The plan is to have several pieces of infrastructure.

• Cloudflare - directs traffic from my domain down a tunnel into a Pi ...

I've recently setup a home server array using Raspberry Pis I've impulsively purchased over the years.

Currently I have the following setup:

• A Pi with Postgres database
• A pi with a network attached storage device
• A Cloudflare tunnel to a Pi

It's Pi running the Cloudflare tunnel I also would like to act as a reverse proxy. This should allow the proxy to direct traffic to the correct place within the home network.

2. Prerequisites

• Raspberry Pi running Ubuntu Server (proxy Pi)
• Static IP set for the proxy Pi
• Other Pis or home servers running web apps (e.g., at ...

Cloudflare Tunnel (formerly Argo Tunnel) allows you to expose your Raspberry Pi 5 services to the internet without port forwarding , without exposing your IP , and with automatic HTTPS —all for free.

This guide walks you through setting up a Cloudflare Tunnel to access a web service running on your Raspberry Pi 5 at home.

🔧 Prerequisites

To get started, you'll need:

• A Cloudflare account (free tier is sufficient)
• A domain name added to Cloudflare (e.g. yourdomain.com )
• A Raspberry Pi 5 with Ubuntu (or compatible Linux)
• A running web service (e.g., Flask app, Nginx, Node, etc.)
• Optional: Static IP ...

This guide outlines steps to turn a Raspberry Pi 5 with 16 GB RAM into a production-ready PostgreSQL server using Ubuntu Server.

1. Prerequisites

• Raspberry Pi 5 (4/8/16 GB RAM)
• Ubuntu Server 24.04 or newer installed
• External SSD for data storage (recommended)
• Static IP configured
• SSH access

2. System Preparation

Update and Upgrade

sudo apt update && sudo apt upgrade -y

Set Hostname and Timezone

sudo hostnamectl set-hostname postgres-server
sudo timedatectl set-timezone America/Chicago

3. Install PostgreSQL

sudo apt install postgresql postgresql-contrib -y

4. Configure PostgreSQL for Production

Tune postgresql.conf

Located at /etc/postgresql/*/main/postgresql.conf ...

As a data engineer and solutions architect I'm often playing with data at home. Lately, I've been gathering image data for training some machine learning algorithms on no-reference automatic image quality assessment (no reference IQA). This problem seems to have been worked on by the group behind the LAION-Aesthetics dataset.

• LAION-Aesthetics

This dataset is massive (at least, to me). The full set is 1.2 billion images. And! I want every one of them. 😂

As a data engineer, I’m more familiar with ETL/ELT pipelines than managing raw filesystems. Let alone managing such a huge file based dataset. After ...

Last time, we set up our local machine for accessing AWS programmatically. This will allow us to use Terraform and Terragrunt to easily create all infrastructure needed for our data warehouse. Now, let's set up Terragrunt and Terraform.

Install Terraform

Navigate to the Terraform downloads page:

• Terraform Download

After installing Terraform enter the following in the terminal:

terraform --version

You should be greeted with output similar to:

Terraform v1.2.8
on darwin_arm64

Install Terragrunt

Terragrunt is a thin wrapper for Terraform, having a few additional tools for managing IaC projects.

Download and install it:

• Terragrunt Downloads
• Terragrunt Package Manager ...

Before we can begin creating infrastructure through tools like Terraform and the Serverless Framework, we need to set up an AWS account and credentials for accessing AWS through the AWS CLI . The AWS CLI will allow us to easily set up programmatic access to AWS, which is necessary to use Terraform and the Serverless Framework to rapidly deploy needed infrastructure.

Creating an AWS Account

Before beginning into AWS, let me warn you: Stuff can get expensive. Please exercise great caution, as leaving the wrong resource on can lead to a heft bill overnight.

To create an account visit: * Instructions for ...

The Reason

I am the lead data engineer at Bitfocus , the best SaaS provider for homeless management information systems ( HMIS ).

For many reasons, I've advocated with our executives to switch our analytics flow to use a data warehouse. Strangely, it worked. I'm now positioned to design the beast. This series will be my humble attempt to journal everything I learn along the way.

The Plan

I've decided it would be better if I learned how to build an analytics warehouse on my own, before committing our entire business to an untested technology stack. Of course, I needed a project similar ...