[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"$f29ilx29gbvzqp":3},{"_id":4,"slug":5,"title":6,"subtitle":7,"kind":8,"cards":9,"tags":58,"categories":60,"source":62,"lang":65,"author":66,"audioState":69,"stats":70,"publishedAt":73,"renderer":74},"6abb0e00ca21c797c7e9be43","building-an-etl-pipeline-with-python-docker-and-postgresql-a-f58b84fa","Building an ETL Pipeline with Python, Docker, and PostgreSQL (And Debugging the Real Errors)","Most ETL tutorials show a perfect, frictionless flow.","news",[10,14,19,24,29,33,38,43,48,53],{"headline":11,"body":12,"imageUrl":13,"sourceImageUrl":13},"Building an ETL Pipeline with Python, Docker, and PostgreSQL (And Debugging the…","Most ETL tutorials show a perfect, frictionless flow. The reality? My pipeline turned into a festival of KeyError's, outdated schemas, and API payload typos.","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1200,height=627,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fs196bsixpmfof4ccfzfa.png",{"headline":15,"body":16,"imageUrl":17,"images":18},"In this article, I will walk you through","In this article, I will walk you through a complete Extrac -> transform -> Load pipeline using Python, Docker, and PostgreSqul. But more importantly, I'll share the real errors I ran into anh how I debugged them. Because calmly reading tracebacks is the only realible way to fix data pipelines when they inevitably fail. GitHub API -> extract.py -> transform.py -> load.py -> PostgreSQL (Docker) Extract: Paginated fetch of issues from a public repository via the GitHub REST API. Transform: Normalizes each issue into a flat record and calculates the hours it took to close.","\u002Fapi\u002Fmedia\u002Fposts\u002Fbuilding-an-etl-pipeline-with-python-docker-and-postgresql-a-f58b84fa\u002F1.webp",{"local":17},{"headline":20,"body":21,"imageUrl":22,"images":23},"Load: Creates the database table if it doesn't","Load: Creates the database table if it doesn't exist ans upserts by ID, so re-running the pipeline updates rows instead of duplicating them. psycopg (v3) to connect to Postgres python-dotenv for configuration management PostgreSQL running in Docker Compose","\u002Fapi\u002Fmedia\u002Fposts\u002Fbuilding-an-etl-pipeline-with-python-docker-and-postgresql-a-f58b84fa\u002F2.webp",{"local":22},{"headline":25,"body":26,"imageUrl":27,"images":28},"Windows + Python 3.14 Note: If you are","Windows + Python 3.14 Note: If you are on Windows, use psycopg[binary] in your requirements instead of psycopg2-binary. The latter often fails to compile cleanly on this combination.","\u002Fapi\u002Fmedia\u002Fposts\u002Fbuilding-an-etl-pipeline-with-python-docker-and-postgresql-a-f58b84fa\u002F3.webp",{"local":27},{"headline":25,"body":30,"imageUrl":31,"images":32},"Windows + Python 3.14 Note: If you are on Windows, use psycopg[binary] in your requirements instead of psycopg2-binary. The latter often fails to compile cleanly on this combination. 2. Spinning up Postgres with Docker First let's get our database running. Using Docker Compose keeps our local enviroment clean. 3. Configuration via .env Create your .env file with the necessary credentials:","\u002Fapi\u002Fmedia\u002Fposts\u002Fbuilding-an-etl-pipeline-with-python-docker-and-postgresql-a-f58b84fa\u002F4.webp",{"local":31},{"headline":34,"body":35,"imageUrl":36,"images":37},"The Common Trap: KeyError: GitHUB_REPO **The Lesson:** __If","The Common Trap: KeyError: GitHUB_REPO **The Lesson:** __If you forget to create.envfrom.env.example, or if you forget callload_dotenv()inmain.py`, the script blows up immediately. Always chechks your enviroment variables first when a pipeline fails at startup.__ 4. Extract: Pulling the Issues We use the request library to handle pagination from the GitHub API.","\u002Fapi\u002Fmedia\u002Fposts\u002Fbuilding-an-etl-pipeline-with-python-docker-and-postgresql-a-f58b84fa\u002F5.webp",{"local":36},{"headline":39,"body":40,"imageUrl":41,"images":42},"`python extract.py import os import logging import requests","`python extract.py import os import logging import requests from tenacity import retry, stop_after_attempt, wait_exponential logger = logging.getLogger(name) GITHUB_API = \"https:\u002F\u002Fapi.github.com\"","\u002Fapi\u002Fmedia\u002Fposts\u002Fbuilding-an-etl-pipeline-with-python-docker-and-postgresql-a-f58b84fa\u002F6.webp",{"local":41},{"headline":44,"body":45,"imageUrl":46,"images":47},"def _headers(): token = os.environ.get(\"GITHUB_TOKEN\") headers = {\"Accept\"","def _headers(): token = os.environ.get(\"GITHUB_TOKEN\") headers = {\"Accept\": \"application\u002Fvnd.github+json\"} if token: headers[\"Authorization\"] = f\"Bearer {token}\" return headers","\u002Fapi\u002Fmedia\u002Fposts\u002Fbuilding-an-etl-pipeline-with-python-docker-and-postgresql-a-f58b84fa\u002F7.webp",{"local":46},{"headline":49,"body":50,"imageUrl":51,"images":52},"@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2…","@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10)) def _get(url, params=None): response = requests.get(url, headers=_headers(), params=params, timeout=10) response.raise_for_status() return response","\u002Fapi\u002Fmedia\u002Fposts\u002Fbuilding-an-etl-pipeline-with-python-docker-and-postgresql-a-f58b84fa\u002F8.webp",{"local":51},{"headline":54,"body":55,"imageUrl":56,"images":57},"def fetch_issues(repo: str, max_pages: int = 5): page","def fetch_issues(repo: str, max_pages: int = 5): page = 1 while page \u003C= max_pages: logger.info(\"Fetching %s issues page %d\", repo, page) resp = _get( f\"{GITHUB_API}\u002Frepos\u002F{repo}\u002Fissues\", params={\"state\": \"all\", \"per_page\": 100, \"page\": page}, ) batch = resp.json() if not batch: break for item in batch: yield item page += 1 ` 5. Transform: Normalizing and Computing Metrics This is what a standard GitHub API response looks like for an issue","\u002Fapi\u002Fmedia\u002Fposts\u002Fbuilding-an-etl-pipeline-with-python-docker-and-postgresql-a-f58b84fa\u002F9.webp",{"local":56},[59],"dev",[61],"Technology",{"name":63,"url":64},"Dev.to","https:\u002F\u002Fdev.to\u002Fwhoismarce\u002Fbuilding-an-etl-pipeline-with-python-docker-and-postgresql-and-debugging-the-real-errors-1kca","en",{"handle":67,"displayName":68},"spots","Spots","queued",{"views":71,"likes":72,"saves":72,"shares":72,"completions":72,"opens":72,"skips":72,"depthSum":72},3,0,"2026-09-29T01:01:52.922Z","local"]